A Comparative Study of Time Series Anomaly Detection Models for Industrial Control Systems

Sensors - Tập 23 Số 3 - Trang 1310
Bedeuro Kim1, Mohsen Ali Alawami1, Eun‐Soo Kim1, Sanghak Oh1, Jeongyong Park2, Hyoungshick Kim1
1Department of Electrical and Computer Engineering, Sungkyunkwan University, 2066 Seobu-ro, Jangan-gu, Suwon-si 16419, Gyeonggi-do, Republic of Korea
2Department of Computer Science and Engineering, Sungkyunkwan University, 2066 Seobu-ro, Jangan-gu, Suwon-si 16419, Gyeonggi-do, Republic of Korea

Tóm tắt

Anomaly detection has been known as an effective technique to detect faults or cyber-attacks in industrial control systems (ICS). Therefore, many anomaly detection models have been proposed for ICS. However, most models have been implemented and evaluated under specific circumstances, which leads to confusion about choosing the best model in a real-world situation. In other words, there still needs to be a comprehensive comparison of state-of-the-art anomaly detection models with common experimental configurations. To address this problem, we conduct a comparative study of five representative time series anomaly detection models: InterFusion, RANSynCoder, GDN, LSTM-ED, and USAD. We specifically compare the performance analysis of the models in detection accuracy, training, and testing times with two publicly available datasets: SWaT and HAI. The experimental results show that the best model results are inconsistent with the datasets. For SWaT, InterFusion achieves the highest F1-score of 90.7% while RANSynCoder achieves the highest F1-score of 82.9% for HAI. We also investigate the effects of the training set size on the performance of anomaly detection models. We found that about 40% of the entire training set would be sufficient to build a model producing a similar performance compared to using the entire training set.

Từ khóa


Tài liệu tham khảo

Nawrocki, M., Schmidt, T.C., and Wählisch, M. (2020, January 20–24). Uncovering Vulnerable Industrial Control Systems from the Internet Core. Proceedings of the IEEE/IFIP Network Operations and Management Symposium, Budapest, Hungary.

Barbieri, G., Conti, M., Tippenhauer, N.O., and Turrin, F. (2020). Sorry, Shodan is not Enough! Assessing ICS Security via IXP Network Traffic Analysis. arXiv.

Di Pinto, A., Dragoni, Y., and Carcano, A. (2018, January 4–9). TRITON: The First ICS Cyber Attack on Safety Instrument Systems. Proceedings of the Black Hat USA, Las Vegas, NV, USA.

Sha, 2018, On Security Challenges and Open Issues in Internet of Things, Future Gener. Comput. Syst., 83, 326, 10.1016/j.future.2018.01.059

Lab, K. (1997). Threat Landscape for Industrial Automation Systems in the Second Half of 2016, AO Kaspersky Lab. Technical Report.

Carcano, 2011, A Multidimensional Critical State Analysis for Detecting Intrusions in SCADA Systems, IEEE Trans. Ind. Inform., 7, 179, 10.1109/TII.2010.2099234

Zhou, 2015, Design and Analysis of Multimodel-Based Anomaly Intrusion Detection Systems in Industrial Process Automation, IEEE Trans. Syst. Man Cybern. Syst., 45, 1345, 10.1109/TSMC.2015.2415763

Clarke, E.M., and Zuliani, P. (2011, January 11–14). Statistical Model Checking for Cyber-Physical Systems. Proceedings of the International Symposium on Automated Technology for Verification and Analysis, Taipei, Taiwan.

Ponomarev, 2015, Industrial Control System Network Intrusion Detection by Telemetry Analysis, IEEE Trans. Dependable Secur. Comput., 13, 252, 10.1109/TDSC.2015.2443793

Zhang, 2019, Multilayer Data-Driven Cyber-Attack Detection System for Industrial Control Systems Based on Network, System, and Process Data, IEEE Trans. Ind. Inform., 15, 4362, 10.1109/TII.2019.2891261

MR, 2020, A Multilayer Perceptron Model for Anomaly Detection in Water Treatment Plants, Int. J. Crit. Infrastruct. Prot., 31, 100393, 10.1016/j.ijcip.2020.100393

Audibert, J., Michiardi, P., Guyard, F., Marti, S., and Zuluaga, M.A. (2020, January 6–10). USAD: Unsupervised Anomaly Detection on Multivariate Time Series. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Virtual.

Li, Z., Zhao, Y., Han, J., Su, Y., Jiao, R., Wen, X., and Pei, D. (2021, January 14–18). Multivariate Time Series Anomaly Detection and Interpretation using Hierarchical Inter-Metric and Temporal Embedding. Proceedings of the ACM SIGKDD Conference on Knowledge Discovery & Data Mining, Virtual.

Abdulaal, A., Liu, Z., and Lancewicki, T. (2021, January 14–18). Practical Approach to Asynchronous Multivariate Time Series Anomaly Detection and Localization. Proceedings of the ACM SIGKDD Conference on Knowledge Discovery & Data Mining, Virtual.

Deng, A., and Hooi, B. (2021, January 2–9). Graph Neural Network-Based Anomaly Detection in Multivariate Time Series. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual.

Malhotra, P., Ramakrishnan, A., Anand, G., Vig, L., Agarwal, P., and Shroff, G. (2016). LSTM-based Encoder-Decoder for Multi-sensor Anomaly Detection. arXiv.

Robles-Durazno, A., Moradpoor, N., McWhinnie, J., and Russell, G. (2018, January 11–12). A supervised energy monitoring-based machine learning approach for anomaly detection in a clean water supply system. Proceedings of the 2018 International Conference on Cyber Security and Protection of Digital Services (Cyber Security), Glasgow, UK.

Mathur, A.P., and Tippenhauer, N.O. (2016, January 11). SWaT: A Water Treatment Testbed for Research and Training on ICS Security. Proceedings of the International Workshop on Cyber-Physical Systems for Smart Water Networks, Vienna, Austria.

Shin, H.K., Lee, W., Yun, J.H., and Min, B.G. (2021, January 9). Two ICS Security Datasets and Anomaly Detection Contest on the HIL-based Augmented ICS Testbed. Proceedings of the Cyber Security Experimentation and Test Workshop, Virtual.

Fu, 2011, A Review on Time Series Data Mining, Eng. Appl. Artif. Intell., 24, 164, 10.1016/j.engappai.2010.09.007

Conde, 2021, A Review on Outlier/Anomaly Detection in Time Series Data, ACM Comput. Surv., 54, 1

Borges, 2021, Anomaly Detection in Time Series, Trans. Large-Scale-Data Knowl. Centered Syst., 54, 1

Inoue, J., Yamagata, Y., Chen, Y., Poskitt, C.M., and Sun, J. (2017, January 18–21). Anomaly Detection for a Water Treatment System using Unsupervised Machine Learning. Proceedings of the IEEE International Conference on Data Mining Workshops, New Orleans, LA, USA.

Firoozjaei, 2022, An evaluation framework for industrial control system cyber incidents, Int. J. Crit. Infrastruct. Prot., 36, 100487, 10.1016/j.ijcip.2021.100487

Karnouskos, S. (2011, January 7–10). Stuxnet worm impact on industrial cyber-physical system security. Proceedings of the IECON 2011—37th Annual Conference of the IEEE Industrial Electronics Society, Melbourne, Australia.

Lee, R.M., Assante, M., and Conway, T. (2017). Crashoverride: Analysis of the Threat to Electric Grid Operations, Dragos Inc.

Geiger, M., Bauer, J., Masuch, M., and Franke, J. (2020, January 8–11). An analysis of black energy 3, Crashoverride, and Trisis, three malware approaches targeting operational technology systems. Proceedings of the 2020 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), Vienna, Austria.

Robles-Durazno, A., Moradpoor, N., McWhinnie, J., and Russell, G. (2019, January 16–19). WaterLeakage: A stealthy malware for data exfiltration on industrial control systems using visual channels. Proceedings of the 2019 IEEE 15th International Conference on Control and Automation (ICCA), Edinburgh, UK.

Chen, Q., Zhou, M., Cai, Z., and Su, S. (2022, January 15–17). Compliance Checking Based Detection of Insider Threat in Industrial Control System of Power Utilities. Proceedings of the 2022 7th Asia Conference on Power and Electrical Engineering (ACPEE), Hangzhou, China.

Giraldo, 2018, A survey of physics-based attack detection in cyber-physical systems, ACM Comput. Surv. CSUR, 51, 1

Komninos, 2014, Survey in smart grid and smart home security: Issues, challenges and countermeasures, IEEE Commun. Surv. Tutorials, 16, 1933, 10.1109/COMST.2014.2320093

Tan, 2016, Survey of security advances in smart grid: A data driven approach, IEEE Commun. Surv. Tutorials, 19, 397, 10.1109/COMST.2016.2616442

Cintuglu, 2016, A survey on smart grid cyber-physical system testbeds, IEEE Commun. Surv. Tutorials, 19, 446, 10.1109/COMST.2016.2627399

He, 2016, Cyber-physical attacks and defences in the smart grid: A survey, IET Cyber Phys. Syst. Theory Appl., 1, 13, 10.1049/iet-cps.2016.0019

Rushanan, M., Rubin, A.D., Kune, D.F., and Swanson, C.M. (2014, January 18–21). Sok: Security and privacy in implantable medical devices and body area networks. Proceedings of the 2014 IEEE Symposium on Security and Privacy, Berkeley, CA, USA.

AlTawy, 2016, Security tradeoffs in cyber physical systems: A case study survey on implantable medical devices, IEEE Access, 4, 959, 10.1109/ACCESS.2016.2521727

Kocabas, 2016, Emerging security mechanisms for medical cyber physical systems, IEEE/ACM Trans. Comput. Biol. Bioinform., 13, 401, 10.1109/TCBB.2016.2520933

Han, 2014, Intrusion detection in cyber-physical systems: Techniques and challenges, IEEE Syst. J., 8, 1052, 10.1109/JSYST.2013.2257594

Mitchell, 2014, A Survey of Intrusion Detection Techniques for Cyber-Physical Systems, ACM Comput. Surv., 46, 1, 10.1145/2542049

Wang, 2016, Recent advances on filtering and control for cyber-physical systems under security and resource constraints, J. Frankl. Inst., 353, 2451, 10.1016/j.jfranklin.2016.04.011

He, H., Maple, C., Watson, T., Tiwari, A., Mehnen, J., Jin, Y., and Gabrys, B. (2016, January 24–29). The security challenges in the IoT enabled cyber-physical systems and opportunities for evolutionary computing & other computational intelligence. Proceedings of the 2016 IEEE Congress on Evolutionary Computation (CEC), Vancouver, BC, Canada.

Kayan, 2022, Cybersecurity of industrial cyber-physical systems: A review, ACM Comput. Surv. CSUR, 54, 1, 10.1145/3510410

Nedeljkovic, 2022, CNN based method for the development of cyber-attacks detection algorithms in industrial control systems, Comput. Secur., 114, 102585, 10.1016/j.cose.2021.102585

Umer, 2022, Machine learning for intrusion detection in industrial control systems: Applications, challenges, and recommendations, Int. J. Crit. Infrastruct. Prot., 2022, 100516, 10.1016/j.ijcip.2022.100516

Wang, 2022, Abnormal detection technology of industrial control system based on transfer learning, Appl. Math. Comput., 412, 126539

Wang, 2017, A novel data analytical approach for false data injection cyber-physical attack mitigation in smart grids, IEEE Access, 5, 26022, 10.1109/ACCESS.2017.2769099

Junejo, K.N., and Yau, D. (2016, January 14–15). Data driven physical modelling for intrusion detection in cyber physical systems. Proceedings of the Singapore Cyber-Security Conference (SG-CRC) 2016, Singapore.

Elgendi, 2019, Protecting cyber physical systems using a learned MAPE-K model, IEEE Access, 7, 90954, 10.1109/ACCESS.2019.2927037

Ahmed, C.M., Ochoa, M., Zhou, J., Mathur, A.P., Qadeer, R., Murguia, C., and Ruths, J. (2018, January 4–8). Noiseprint: Attack detection using sensor and process noise fingerprint in cyber physical systems. Proceedings of the 2018 on Asia Conference on Computer and Communications Security, Incheon, Republic of Korea.

Li, D., Chen, D., Jin, B., Shi, L., Goh, J., and Ng, S.K. (2019, January 17–19). MAD-GAN: Multivariate Anomaly Detection for Time Series Data with Generative Adversarial Networks. Proceedings of the International Conference on Artificial Neural Networks, Munich, Germany.

Su, Y., Zhao, Y., Niu, C., Liu, R., Sun, W., and Pei, D. (2019, January 4–8). Robust Anomaly Detection for Multivariate Time Series through Stochastic Recurrent Neural Network. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Anchorage, AK, USA.

Breiman, 1996, Bagging Predictors, Mach. Learn., 24, 123, 10.1007/BF00058655

LeCun, 2015, Deep Learning, Nature, 521, 436, 10.1038/nature14539

Prashanth, 2022, Optimal Feature Selection Based on Evolutionary Algorithm for Intrusion Detection, SN Comput. Sci., 3, 1, 10.1007/s42979-022-01325-4

Chen, 2015, Xgboost: Extreme gradient boosting, R Package Version 0.4-2, 1, 1

Zong, B., Song, Q., Min, M.R., Cheng, W., Lumezanu, C., Cho, D., and Chen, H. (May, January 30). Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada.

Hundman, K., Constantinou, V., Laporte, C., Colwell, I., and Soderstrom, T. (2018, January 19–23). Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK.

Ikeda, Y., Tajiri, K., Nakano, Y., Watanabe, K., and Ishibashi, K. (2018). Estimation of Dimensions Contributing to Detected Anomalies with Variational Autoencoders. arXiv.

Zhai, S., Cheng, Y., Lu, W., and Zhang, Z. (2016, January 19–24). Deep Structured Energy Based Models for Anomaly Detection. Proceedings of the International Conference on Machine Learning, New York, NY, USA.

Zhang, C., Song, D., Chen, Y., Feng, X., Lumezanu, C., Cheng, W., Ni, J., Zong, B., Chen, H., and Chawla, N.V. (February, January 27). A Deep Neural Network for Unsupervised Anomaly Detection and Diagnosis in Multivariate Time Series Data. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.

Shitharth, 2017, An enhanced optimization based algorithm for intrusion detection in SCADA network, Comput. Secur., 70, 16, 10.1016/j.cose.2017.04.012

Estabrooks, A., and Japkowicz, N. (2001, January 13–15). A Mixture-of-Experts Framework for Learning from Imbalanced Data Sets. Proceedings of the International Symposium on Intelligent Data Analysis, Cascais, Portugal.

Johnson, 2019, Survey on Deep Learning with Class Imbalance, J. Big Data, 6, 1, 10.1186/s40537-019-0192-5

Macas, M., and Chunming, W. (2019, January 23). Enhanced Cyber-Physical Security through Deep Learning Techniques. Proceedings of the CPS Summer School PhD Workshop, Alghero, Italy.

Thông tin
Thông tin xuất bản

Sensors

Tập 23 Số 3

1310

Thông tin tác giả