Dirichlet Graph Convolution Coupled Neural Differential Equation for Spatio-temporal Time Series Prediction

Springer Science and Business Media LLC - Tập 55 - Trang 12347-12366 - 2023
Qipeng Wang1, Min Han2,3
1Faculty of Electronic Information and Electrical Engineering, Dalian University of Technology, Dalian, China
2Key Laboratory of Intelligent Control and Optimization for Industrial Equipment of Ministry of Education, Dalian University of Technology, Dalian, China
3Professional Technology Innovation Center of Distributed Control for Industrial Equipment of Liaoning Province, Dalian University of Technology, Dalian, China

Tóm tắt

In recent years, multivariate time series prediction has attracted extensive research interests. However, the dynamic changes of the spatial topology and the temporal evolution of multivariate variables bring great challenges to the spatio-temporal time series prediction. In this paper, a novel Dirichlet graph convolution module is introduced to automatically learn the spatio-temporal representation, and we combine graph attention (GAT) and neural differential equation (NDE) based on nonlinear state transition to model spatio-temporal state evolution of nonlinear systems. Specifically, the spatial topology is revealed by the cosine similarity of node embeddings. The use of multi-layer Dirichlet graph convolution aims to enhance the representation ability of the model while suppressing the phenomenon of over-smoothing or over-separation. The GCN and LSTM-based network is used as the nonlinear operator to model the evolution law of the dynamic system, and the GAT updates the strength of the connection. In addition, the Euler trapezoidal integral method is used to model the temporal dynamics and makes medium and long-term prediction in latent space from the perspective of nonlinear state transition. The proposed model can adaptively mine spatial correlations and discover spatio-temporal dynamic evolution patterns through the coupled NDE, which makes the modeling process more interpretable. Experiment results demonstrate the effectiveness of spatio-temporal dynamic discovery on predictive performance.

Tài liệu tham khảo

Li D, Han M, Wang J (2012) Chaotic time series prediction based on a novel robust echo state network. IEEE Trans Neural Netw Learn Syst 23(5):787–799 Quan H, Srinivasan D, Khosravi A (2014) Short-term load and wind power forecasting using neural network-based prediction intervals. IEEE Trans Neural Netw Learn Syst 25(2):303–315 Pal S, Ma L, Zhang Y, Coates M (2021) RNN with particle flow for probabilistic spatio-temporal forecasting. In: Proceedings of ICML, pp 8336–8348 Spadondesouza G, Hong S, Brandoli B, Matwin S, Rodrigues JF, Sun J (2021) Pay attention to evolution: time series forecasting with deep graph-evolution learning. IEEE Trans Pattern Anal Mach Intell 44:1–17 Liu Y, Zheng Y, Liang Y, Liu S, Rosenblum DS (2016) Urban water quality prediction based on multi-task multi-view learning. In: Proceedings of IJCAI Lv Y, Duan Y, Kang W, Li Z, Wang FY (2015) Traffic flow prediction with big data: a deep learning approach. IEEE Trans Intell Transp Syst 16(2):865–873 Han M, Feng S, Chen CP, Xu M, Qiu T (2018) Structured manifold broad learning system: a manifold perspective for large-scale chaotic time series analysis and prediction. IEEE Trans Knowl Data Eng 31(9):1809–1821 Yu B, Yin H, Zhu Z (2018) Spatio-temporal graph convolutional networks: a deep learning framework for traffic forecasting. In: Proceedings of IJCAI, pp 3634–3640 Song C, Lin Y, Guo S, Wan H (2020) Spatial-temporal synchronous graph convolutional networks: a new framework for spatial-temporal network data forecasting. In: Proceedings of AAAI, vol 34, pp 914–921 Wu Z, Pan S, Chen F, Long G, Zhang C, Yu PS (2021) A comprehensive survey on graph neural networks. IEEE Trans Neural Netw Learn Syst 32(1):4–24 Spinelli I, Scardapane S, Uncini A (2020) Adaptive propagation graph convolutional network. IEEE Trans Neural Netw Learn Syst 32(10):4755–4760 Zhou K, Huang X, Zha D, Chen R, Li L, Choi S-H, Hu X (2021) Dirichlet energy constrained learning for deep graph neural networks. In: Proceedings of NeurIPS Wang X, Zhu M, Bo D, Cui P, Shi C, Pei J (2020) AM-GCN: adaptive multi-channel graph convolutional networks. In: Proceedings of ACM SIGKDD, pp 1243–1253 Shi S, Xie P, Luo X, Qiao K, Wang L, Chen J, Yan B (2022) Adaptive multi-layer contrastive graph neural networks. Neural Process Lett 55:1–20 Wang C, Zhang H, Chen B, Wang D, Wang Z, Zhou M (2020) Deep relational topic modeling via graph Poisson gamma belief network. In: Proceedings of NeurIPS, pp 488–500 Huang Z, Sun Y, Wang W (2021) Coupled graph ode for learning interacting system dynamics. In: Proceedings of ACM SIGKDD, pp 705–715 Lai G, Chang W-C, Yang Y, Liu H (2018) Modeling long-and short-term temporal patterns with deep neural networks. In: Proceedings of ACM SIGIR, pp 95–104 Jalali SMJ, Ahmadian S, Kavousi-Fard A, Khosravi A, Nahavandi S (2021) Automated deep CNN-LSTM architecture design for solar irradiance forecasting. IEEE Trans Syst Man Cybern Syst 52(1):54–65 Hu J, Wang X, Zhang Y, Zhang D, Zhang M, Xue J (2020) Time series prediction method based on variant LSTM recurrent neural network. Neural Process Lett 52:1485–1500 Chen TQ, Rubanova Y, Bettencourt J, Duvenaud DK (2018) Neural ordinary differential equations. In: Proceedings of NeurIPS Zhao L, Song Y, Zhang C, Liu Y, Wang P, Lin T, Deng M, Li H (2019) T-GCN: a temporal graph convolutional network for traffic prediction. IEEE Trans Intell Transp Syst 21(9):3848–3858 Bruna J, Zaremba W, Szlam A, LeCun Y (2014) Spectral networks and deep locally connected networks on graphs. In: Proceedings of ICLR, pp 1–14 Kipf TN, Welling M (2017) Semi-supervised classification with graph convolutional networks. In: Proceedings of ICLR Xu Z, Kang Y, Cao Y, Li Z (2020) Spatiotemporal graph convolution multifusion network for urban vehicle emission prediction. IEEE Trans Neural Netw Learn Syst 32(8):3342–3354 Li L, Zhu H, Wen L, Lan W, Yang Z (2021) An approach of combining convolution neural network and graph convolution network to predict the progression of myopia. Neural Process Lett 55:1–11 Huang R, Huang C, Liu Y, Dai G, Kong W (2020) LSGCN: long short-term traffic prediction with graph convolutional networks. In: Proceedings of IJCAI, pp 2355–2361 Li R, Wang S, Zhu F, Huang J (2018) Adaptive graph convolutional neural networks. In: Proceedings of AAAI, vol 32 He Y, Wang C, Zhang H, Chen B, Zhou M (2022) A variational edge partition model for supervised graph representation learning. In: Proceedings of NeurIPS, pp 12339–12351 Sareminia S (2022) A support vector based hybrid forecasting model for chaotic time series: spare part consumption prediction. Neural Process Lett 55:1–17 Zhou H, Zhang S, Peng J, Zhang S, Li J, Xiong H, Zhang W (2021) Informer: beyond efficient transformer for long sequence time-series forecasting. In: Proceedings of AAAI, pp 1–9 Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A.N, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: Proceedings of NeurIPS, pp 5998–6008 Bandara K, Bergmeir C, Hewamalage H (2020) LSTM-MSNET: leveraging forecasts on sets of related time series with multiple seasonal patterns. IEEE Trans Neural Netw Learn Syst 32(4):1586–1599 Mei J, Ren W, Song Y (2021) A unified framework for adaptive leaderless consensus of uncertain multiagent systems under directed graphs. IEEE Trans Autom Control 66(12):6179–6186 Erichson NB, Muehlebach M, Mahoney MW (2019) Physics-informed autoencoders for Lyapunov-stable fluid flow prediction. In: Proceedings of NeurIPS, pp 1–6 Zhang Z, Cui P, Pei J, Wang X, Zhu W (2021) Eigen-GNN: a graph structure preserving plug-in for GNNs. IEEE Trans Knowl Data Eng 1–12 Li X, Wen C, Zou Y (2020) Adaptive backstepping control for fractional-order nonlinear systems with external disturbance and uncertain parameters using smooth control. IEEE Trans Syst Man Cybern Systms 51(12):7860–7869 Wang X, Zhu M, Bo D, Cui P, Shi C, Pei J (2020) AM-GCN: adaptive multi-channel graph convolutional networks. In: Proceedings of ACM SIGKDD, pp 1243–1253 Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780 Abramowitz M, Stegun IA (1964) Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables. U.S. Department of Commerce, National Bureau of Standards Applied Mathematics Series Lorenz EN (1996) Predictability: a problem partly solved. In: Proceedings of seminar on predictability Oreshkin BN, Carpov D, Chapados N, Bengio Y (2020) N-beats: neural basis expansion analysis for interpretable time series forecasting. In: Proceedings of ICLR Chung J, Gulcehre C, Cho K, Bengio Y (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. In: Proceedings of NeurIPS Lai G, Chang W-C, Yang Y, Liu H (2018) Modeling long-and short-term temporal patterns with deep neural networks. In: Proceedings of ACM SIGIR, pp 95–104 Wu Z, Pan S, Long G, Jiang J, Zhang C (2019) Graph wavenet for deep spatial-temporal graph modeling. In: Proceedings of IJCAI Wang X, Ma Y, Wang Y, Jin W, Wang X, Tang J, Jia C, Yu J (2020) Traffic flow prediction via spatial temporal graph neural network. In: Proceedings of WWW, pp 1082–1092 Lan S, Ma Y, Huang W, Wang W, Yang H, Li P (2022) DSTAGNN: dynamic spatial-temporal aware graph neural network for traffic flow forecasting. In: Proceedings of ICML, pp 11906–11917 Wu H, Xu J, Wang J, Long M (2021) Autoformer: decomposition transformers with auto-correlation for long-term series forecasting. In: Proceedings of NeurIPS, pp 22419–22430