Sensor data quality: a systematic review

Gubbi J, Buyya R, Marusic S, Palaniswami M. Internet of Things (IoT): a vision, architectural elements, and future directions. Future Gener Comput Syst. 2013;29(7):1645–60. https://doi.org/10.1016/j.future.2013.01.010. Cisco: Cisco global cloud index: Forecast and methodology, 2016-2021. Whitepaper c11-738085, Cisco Systems Inc., San Jose, CA (2018). https://www.cisco.com/c/en/us/solutions/collateral/service-provider/global-cloud-index-gci/white-paper-c11-738085.pdf Zhang P. Advanced industrial control technology. Oxford: William Andrew Publishing; 2010. https://doi.org/10.1016/B978-1-4377-7807-6.10003-8. Wang RY, Strong DM. Beyond accuracy: what data quality means to data consumers. J Manag Inform Syst. 1996;12(4):5–33. Karkouch A, Mousannif H, Al Moatassime H, Noel T. Data quality in internet of things: a state-of-the-art survey. J Netw Comput Appl. 2016;73:57–81. https://doi.org/10.1016/j.jnca.2016.08.002. Christ M, Krumeich J, Kempa-Liehr AW. Integrating predictive analytics into complex event processing by using conditional density estimations. In: IEEE 20th international enterprise distributed object computing workshop (EDOCW). In: IEEE computer society, Los Alamitos, CA, USA; 2016. pp. 1–8. https://doi.org/10.1109/EDOCW.2016.7584363. Zhang H, Liu J, Pang A-C. A Bayesian network model for data losses and faults in medical body sensor networks. Comput Netw. 2018;143:166–75. https://doi.org/10.1016/j.comnet.2018.07.009. Ye J, Stevenson G, Dobson S. Detecting abnormal events on binary sensors in smart home environments. Pervasive Mobile Comput. 2016;33:32–49. https://doi.org/10.1016/j.pmcj.2016.06.012. Li Y, Parker LE. Nearest neighbor imputation using spatial-temporal correlations in wireless sensor networks. Inform Fusion. 2014;15:64–79. https://doi.org/10.1016/j.inffus.2012.08.007. Cheng R, Chen J, Xie X. Cleaning uncertain data with quality guarantees. Proc VLDB Endow. 2008;1(1):722–35. https://doi.org/10.14778/1453856.1453935. Ray PP. A survey on Internet of Things architectures. J King Saud Univ Comput Inform Sci. 2018;30(3):291–319. Lin J, Yu W, Zhang N, Yang X, Zhang H, Zhao W. A Survey on Internet of Things: architecture, enabling technologies, security and privacy, and applications. IEEE Intern Things J. 2017;4(5):1125–42. https://doi.org/10.1109/JIOT.2017.2683200. Ahmed E, Yaqoob I, Hashem IAT, Khan I, Ahmed AIA, Imran M, Vasilakos AV. The role of big data analytics in Internet of Things. Comput Netw. 2017;129:459–71. https://doi.org/10.1016/j.comnet.2017.06.013. Li Y, Chen J, Feng L. Dealing with uncertainty: a survey of theories and practices. IEEE Trans Knowl Data Eng. 2013;25(11):2463–82. https://doi.org/10.1109/TKDE.2012.179. Prathiba B, Sankar KJ, Sumalatha V. Enhancing the data quality in wireless sensor networks - a review. In: 2016 international conference on automatic control and dynamic optimization techniques (ICACDOT). 2016;448–454. https://doi.org/10.1109/ICACDOT.2016.7877626. Kofod-Petersen A. How to do a structured literature review in computer science. (2015). Silva R, Neiva F. Systematic literature review in computer science—a practical guide. (2016). https://doi.org/10.13140/RG.2.2.35453.87524. PRISMA: PRISMA—transparent reporting of systematic reviews and meta-analyses (2015). http://www.prisma-statement.org/ Accessed 08 Jan 2019. Blei DM, Lafferty JD. Topic models. In: Ashok N, Srivastava MS, editors. Text mining. Classification, clustering, and applications. Chapman and Hall/CRC: New York; 2009. p. 71–93. Zhai C. Statistical language models for information retrieval. Synth Lectures Human Lang Technol. 2008;1(1):1–41. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E. Scikit-learn: machine learning in Python. J Mach Learn Res. 2011;12:2825–30. Chow LS, Paramesran R. Review of medical image quality assessment. Biomed Sign Process Contr. 2016;27:145–54. https://doi.org/10.1016/j.bspc.2016.02.006. Lapini A, Argenti F, Piva A, Bencini L. Comparison of super-resolution methods for quality enhancement of digital biomedical images. In: 2014 8th International symposium on medical information and communication technology (ISMICT). 2014. https://doi.org/10.1109/ISMICT.2014.6825243. pp. 1–5. Sharma P, Sharma S. An analysis of vision based techniques for quality assessment and enhancement of camera captured document images. In: 2016 6th international conference—cloud system and Big Data engineering (Confluence). 2016. pp. 425–28. https://doi.org/10.1109/CONFLUENCE.2016.7508157. Bamgboye O, Liu X, Cruickshank P. Towards modelling and reasoning about uncertain data of sensor measurements for decision support in smart spaces. In: 2018 IEEE 42nd annual computer software and applications conference (COMPSAC), 2018. pp. 744–49. https://doi.org/10.1109/COMPSAC.2018.10330. Kuka C, Nicklas D. Enriching sensor data processing with quality semantics. In: 2014 IEEE international conference on pervasive computing and communication workshops (PERCOM WORKSHOPS). 2014. pp. 437–42. https://doi.org/10.1109/PerComW.2014.6815246. Dunia R, Joe Qin S, Edgar TF, McAvoy TJ. Use of principal component analysis for sensor fault identification. Comput Chem Eng. 1996;20:713–8. https://doi.org/10.1016/0098-1354(96)00128-7. Moher D, Liberati A, Tetzlaff J, Altman DG, Group TP. Preferred reporting items for systematic reviews and meta-analyses: the prisma statement. PLoS Med. 2009;6(7):1–6. https://doi.org/10.1371/journal.pmed.1000097. Joint Committee Guides Metrology: evaluation of measurement data-guide to the expression of uncertainty in measurement (GUM 2008). 2008. Chen Y, Jiang S, Yang J, Song K, Wang Q. Grey bootstrap method for data validation and dynamic uncertainty estimation of self-validating multifunctional sensors. Chemometr Intell Lab Syst. 2015;146:63–76. https://doi.org/10.1016/j.chemolab.2015.05.003. Feng J, Hajizadeh I, Samadi S, Sevil M, Hobbs N, Brandt R, Lazaro C, Maloney Z, Yu X, Littlejohn E, Quinn L, Cinar A. Hybrid online multi-sensor error detection and functional redundancy for artificial pancreas control systems. IFAC-PapersOnLine. 2018;51(18):138–43. https://doi.org/10.1016/j.ifacol.2018.09.289. Harkat MF, Mourot G, Ragot J. Sensor failure detection of air quality monitoring network. IFAC Proc Vol. 2000;33(11):529–34. https://doi.org/10.1016/S1474-6670(17)37413-X. Abuaitah GR, Wang B. Data-centric anomalies in sensor network deployments: analysis and detection. In: 2012 IEEE 9th international conference on mobile Ad-Hoc and sensor systems (MASS 2012), vol. Supplement. 2012. pp. 1–6. https://doi.org/10.1109/MASS.2012.6708514. Ahmad S, Lavin A, Purdy S, Agha Z. Unsupervised real-time anomaly detection for streaming data. Neurocomputing. 2017;262:134–47. https://doi.org/10.1016/j.neucom.2017.04.070. Bosman HHWJ, Iacca G, Tejada A, Wörtche HJ, Liotta A. Ensembles of incremental learners to detect anomalies in ad hoc sensor networks. Ad Hoc Netw. 2015;35:14–36. https://doi.org/10.1016/j.adhoc.2015.07.013. Bosman HH, Iacca G, Tejada A, Wörtche HJ, Liotta A. Spatial anomaly detection in sensor networks using neighborhood information. Inform Fusion. 2017;33:41–56. https://doi.org/10.1016/j.inffus.2016.04.007. Curiac D-I, Volosencu C. Ensemble based sensing anomaly detection in wireless sensor networks. Exp Syst Appl. 2012;39(10):9087–96. https://doi.org/10.1016/j.eswa.2012.02.036. Dereszynski EW, Dietterich TG. Spatiotemporal models for data-anomaly detection in dynamic environmental monitoring campaigns. ACM Trans Sen Netw. 2011;8(1):3–1336. https://doi.org/10.1145/1993042.1993045. Fawzy A, Mokhtar HMO, Hegazy O. Outliers detection and classification in wireless sensor networks. Egypt Inform J. 2013;14(2):157–64. https://doi.org/10.1016/j.eij.2013.06.001. Hill DJ, Minsker BS. Anomaly detection in streaming environmental sensor data: a data-driven modeling approach. Environ Model Softw. 2010;25(9):1014–22. https://doi.org/10.1016/j.envsoft.2009.08.010. Hou Z, Lian Z, Yao Y, Yuan X. Data mining based sensor fault diagnosis and validation for building air conditioning system. Energy Convers Manag. 2006;47(15):2479–90. https://doi.org/10.1016/j.enconman.2005.11.010. Hu Y, Chen H, Li G, Li H, Xu R, Li J. A statistical training data cleaning strategy for the PCA-based chiller sensor fault detection, diagnosis and data reconstruction method. Energy Build. 2016;112:270–8. https://doi.org/10.1016/j.enbuild.2015.11.066. Huang X-h. Sensor fault diagnosis and reconstruction of engine control system based on autoassociative neural network. Chin J Aeronaut. 2004;17(1):23–7. https://doi.org/10.1016/S1000-9361(11)60198-2. Ibarguengoytia PH, Sucar LE, Vadera S. Real time intelligent sensor validation. IEEE Trans Power Syst. 2001;16(4):770–5. https://doi.org/10.1109/59.962425. Liu H, Chen J, Huang F, Li H. An electric power sensor data oriented data cleaning solution. In: 2017 14th international symposium on pervasive systems, algorithms and networks 2017 11th international conference on frontier of computer science and technology 2017 Third international symposium of creative computing (ISPAN-FCST-ISCC). 2017. pp. 430–5. https://doi.org/10.1109/ISPAN-FCST-ISCC.2017.29. Liu Y, Chen J, Sun Z, Li Y, Huang D. A probabilistic self-validating soft-sensor with application to wastewater treatment. Comput Chem Eng. 2014;71:263–80. https://doi.org/10.1016/j.compchemeng.2014.08.008. Mansouri M, Harkat M-F, Nounou M, Nounou H. Midpoint-radii principal component analysis—based EWMA and application to air quality monitoring network. Chemometr Intell Lab Syst. 2018;175:55–64. https://doi.org/10.1016/j.chemolab.2018.01.016. Rassam MA, Maarof MA, Zainal A. Adaptive and online data anomaly detection for wireless sensor systems. Knowl Syst. 2014;60:44–57. https://doi.org/10.1016/j.knosys.2014.01.003. Sallans B, Bruckner D, Russ G. Statistical model-based sensor diagnostics for automation systems. In: Chávez, M.L., ed. Fieldbus systems and their applications Elsevier: Oxford; 2006. pp. 239–46.https://doi.org/10.1016/B978-008045364-4/50073-3. http://www.sciencedirect.com/science/article/pii/B9780080453644500733. Sharifi R, Langari R. Nonlinear sensor fault diagnosis using mixture of probabilistic PCA models. Mech Syst Sign Process. 2017;85:638–50. https://doi.org/10.1016/j.ymssp.2016.08.028. Solomakhina N, Hubauer T, Lamparter S, Roshchin M, Grimm S. Extending statistical data quality improvement with explicit domain models. In: 2014 12th IEEE international conference on industrial informatics (INDIN). 2014. pp. 720–5. https://doi.org/10.1109/INDIN.2014.6945602. Tsang KM. Sensor data validation using gray models. ISA Trans. 2003;42(1):9–17. https://doi.org/10.1016/S0019-0578(07)60109-8. Tsang KM, Chan WL. Data validation of intelligent sensor using predictive filters and fuzzy logic. Sens Actuat A. 2010;159(2):149–56. https://doi.org/10.1016/j.sna.2010.03.013. Xiao H, Huang D, Pan Y, Liu Y, Song K. Fault diagnosis and prognosis of wastewater processes with incomplete data by the auto-associative neural networks and ARMA model. Chemometr Intell Lab Syst. 2017;161:96–107. https://doi.org/10.1016/j.chemolab.2016.12.009. Liu Y, Daoping H, Zhifu L. A SEVA soft sensor method based on self-calibration model and uncertainty description algorithm. Chemometr Intell Lab Syst. 2013;126:38–49. https://doi.org/10.1016/j.chemolab.2013.04.009. Yu Z, Bedig A, Montalto F, Quigley M. Automated detection of unusual soil moisture probe response patterns with association rule learning. Environ Modell Softw. 2018;105:257–69. https://doi.org/10.1016/j.envsoft.2018.04.001. Zhang Y, Meratnia N, Havinga P. Adaptive and online one-class support vector machine-based outlier detection techniques for wireless sensor networks. In: 2009 international conference on advanced information networking and applications workshops. 2009. pp. 990–5. https://doi.org/10.1109/WAINA.2009.200. Zhang Y, Meratnia N, Havinga PJM. Distributed online outlier detection in wireless sensor networks using ellipsoidal support vector machine. Ad Hoc Netw. 2013;11(3):1062–74. https://doi.org/10.1016/j.adhoc.2012.11.001. Zhao C, Fu Y. Statistical analysis based online sensor failure detection for continuous glucose monitoring in type I diabetes. Chemometr Intell Lab Syst. 2015;144:128–37. https://doi.org/10.1016/j.chemolab.2015.04.001. Yang J, Lin L, Sun Z, Chen Y, Jiang S. Data validation of multifunctional sensors using independent and related variables. Sens Actuat A. 2017;263:76–90. https://doi.org/10.1016/j.sna.2017.05.015. Chok H, Gruenwald L. Spatio-temporal association rule mining framework for real-time sensor network applications. In: Proceedings of the 18th ACM conference on information and knowledge management. CIKM ’09. ACM: New York; 2009. pp. 1761–4. https://doi.org/10.1145/1645953.1646224. Accessed 31 Aug 2018. D’Aniello G, Gaeta M, Hong TP. Effective quality-aware sensor data management. IEEE Trans Emerg Top Comput Intell. 2018;2(1):65–77. https://doi.org/10.1109/TETCI.2017.2782800. Fekade B, Maksymyuk T, Kyryk M, Jo M. Probabilistic recovery of incomplete sensed data in IoT. IEEE Intern Things J. 2017;. https://doi.org/10.1109/JIOT.2017.2730360. Gruenwald L, Chok H, Aboukhamis M. Using data mining to estimate missing sensor data. In: Seventh IEEE international conference on data mining workshops (ICDMW 2007), 2007. pp. 207–12. https://doi.org/10.1109/ICDMW.2007.103. Tang J, Zhang G, Wang Y, Wang H, Liu F. A hybrid approach to integrate fuzzy C-means based imputation method with genetic algorithm for missing traffic volume data estimation. Transport Res C. 2015;51:29–40. https://doi.org/10.1016/j.trc.2014.11.003. Wang Y, Wang J, Li H. An interpolation approach for missing context data based on the time-space relationship and association rule mining. In: 2011 third international conference on multimedia information networking and security, 2011. pp. 623–7. https://doi.org/10.1109/MINES.2011.78. Xu P, Ruan W, Sheng QZ, Gu T, Yao L. Interpolating the missing values for multi-dimensional spatial-temporal sensor data: a tensor SVD approach. In: Proceedings of the 14th EAI international conference on mobile and ubiquitous systems: computing, networking and services. MobiQuitous 2017. pp. 442–51. ACM: New York; 2017. https://doi.org/10.1145/3144457.3144474. Hermans F, Dziengel N, Schiller J. Quality estimation based data fusion in wireless sensor networks. In: 2009 IEEE 6th international conference on mobile adhoc and sensor systems. 2009. pp. 1068–70. https://doi.org/10.1109/MOBHOC.2009.5337006. Alawi A, Choi SW, Martin E, Morris J. Sensor fault identification using weighted combined contribution plots. In: Zhang H-Y, ed. Fault detection, supervision and safety of technical processes 2006. 2007. pp. 908–13. https://doi.org/10.1016/B978-008044485-7/50153-6. http://www.sciencedirect.com/science/article/pii/B9780080444857501536. Smarsly K, Law KH. Decentralized fault detection and isolation in wireless structural health monitoring systems using analytical redundancy. Adv Eng Softw. 2014;73:1–10. https://doi.org/10.1016/j.advengsoft.2014.02.005. Tadić P, Durović Z. Particle filtering for sensor fault diagnosis and identification in nonlinear plants. J Process Control. 2014;24(4):401–9. https://doi.org/10.1016/j.jprocont.2014.02.009. Uren KR, Schoor Gv, Rand CPd, Botha A. An integrated approach to sensor FDI and signal reconstruction in HTGRs—Part I: theoretical framework. Ann Nucl Energy. 2016;87:750–60. https://doi.org/10.1016/j.anucene.2015.06.010. Yu Y, Li H. Virtual in-situ calibration method in building systems. Autom Constr. 2015;59:59–67. https://doi.org/10.1016/j.autcon.2015.08.003. Wang Y, Yang A, Li Z, Wang P, Yang H. Blind drift calibration of sensor networks using signal space projection and Kalman filter. In: 2015 IEEE tenth international conference on intelligent sensors, sensor networks and information processing (ISSNIP). 2015. pp. 1–6. https://doi.org/10.1109/ISSNIP.2015.7106904. Zahedi S, Szczodrak M, Ji P, Mylaraswamy D, Srivastava M, Young R. Tiered architecture for on-line detection, isolation and repair of faults in wireless sensor networks. In: MILCOM 2008–2008 In: IEEE military communications conference. 2008. pp. 1–7. https://doi.org/10.1109/MILCOM.2008.4753634. Omitaomu OA, Protopopescu VA, Ganguly AR. Empirical mode decomposition technique with conditional mutual information for denoising operational sensor data. IEEE Sens J. 2011;11(10):2565–75. https://doi.org/10.1109/JSEN.2011.2142302. Sadıkoglu F, Kavalcıoğlu C. Filtering continuous glucose monitoring signal using Savitzky–Golay filter and simple multivariate thresholding. Proc Comput Sci. 2016;102:342–50. https://doi.org/10.1016/j.procs.2016.09.410. Jäger G, Zug S, Brade T, Dietrich A, Steup C, Moewes C, Cretu AM. Assessing neural networks for sensor fault detection. In: 2014 IEEE international conference on computational intelligence and virtual environments for measurement systems and applications (CIVEMSA). 2014. pp. 70–5. https://doi.org/10.1109/CIVEMSA.2014.6841441. Rahman A, Smith DV, Timms G. A novel machine learning approach toward quality assessment of sensor data. IEEE Sens J. 2014;14(4):1035–47. https://doi.org/10.1109/JSEN.2013.2291855. Richter C. Reliability assessment in everyday-objects based physical-activity sensing using personal information. In: Proceedings of the 8th ACM international conference on pervasive technologies related to assistive environments. PETRA ’15, pp. 39–1394. ACM: New York; 2015. https://doi.org/10.1145/2769493.2769548. Wang P, Gao RX, Tang X, Fan Z. Sensing uncertainty evaluation for product quality. Proc CIRP. 2016;41:706–11. https://doi.org/10.1016/j.procir.2015.12.105. Aggarwal CC. An introduction to outlier analysis. Outlier analysis. Springer: New York; 2013. p. 1–40. https://doi.org/10.1007/978-1-4614-6396-2_1. Ahmad NF, Hoang DB, Phung MH. Robust preprocessing for health care monitoring framework. In: 2009 11th international conference on e-Health networking, applications and services (Healthcom). 2009. pp. 169–74. https://doi.org/10.1109/HEALTH.2009.5406196. Rabatel J, Bringay S, Poncelet P. Anomaly detection in monitoring sensor data for preventive maintenance. Expert Syst Appl. 2011;38(6):7003–15. https://doi.org/10.1016/j.eswa.2010.12.014. Press WH, Teukolsky SA, Vetterling WT, Flannery BP. Numerical recipes. The art of scientific computing. 3rd ed. Cambridge: Cambridge University Press; 2007. Kramer MA. Autoassociative neural networks. Comput Chem Eng. 1992;16(4):313–28. https://doi.org/10.1016/0098-1354(92)80051-A. Hawkins J, Blakeslee S. On intelligence. New York: Times Books; 2004. Numenta: Numenta—Home of the HTM Community (2019). https://numenta.org/. Accessed 08 Jan 2019. Fisher RA. Statistical methods for research workers. In: Kotz S, Johnson NL, editors. Breakthroughs in statistics: methodology and distribution Springer series in statistics. Springer: New York; 1992. p. 66–70. https://doi.org/10.1007/978-1-4612-4380-9_6. Christ M, Braun N, Neuffer J, Kempa-Liehr AW. Time series featuRe extraction on basis of scalable hypothesis tests (tsfresh—a python package). Neurocomputing. 2018;307:72–7. https://doi.org/10.1016/j.neucom.2018.03.067. Deng J-L. Control problems of grey systems. Syst Contr Lett. 1982;1(5):288–94. https://doi.org/10.1016/S0167-6911(82)80025-X. Huang G-B, Zhu Q-Y, Siew C. Extreme learning machine: a new learning scheme of feedforward neural networks. Neural Netw. 2004;2:985–9902. https://doi.org/10.1109/IJCNN.2004.1380068. Ingelrest F, Barrenetxea G, Schaefer G, Vetterli M, Couach O, Parlange M. Sensorscope: application-specific sensor network for environmental monitoring. ACM Trans Sens Netw. 2010;6(2):17. Barrenetxea G. Sensorscope: Sensor Networks for Environmental Monitoring (2018). https://doi.org/10.5281/zenodo.2654726. https://lcav.epfl.ch/research/research-archives/research-archives-communications_and_sensor_networks_archive-html/sensorscope-en/page-145180-en-html/. Accessed 08 May 2019. Madden S. Intel Lab Data (2004). http://db.csail.mit.edu/labdata/labdata.html. Accessed 08 May 2019. Dua D, Graff C. UCI machine learning repository (2017). http://archive.ics.uci.edu/ml Accessed 08 May 2019. University of Southern California: Networked Aquatic Microbial Observing System (NAMOS). http://robotics.usc.edu/~namos/data.html. 2002. Numenta: the numenta anomaly benchmark. 2019. https://github.com/numenta/NAB. Accessed 08 May 2019. of California S. California department of transportation: caltrans performance measurement system; 2019. http://pems.dot.ca.gov/. Accessed 08 May 2019. Timms G, Sharman C, Howell B, McCulloch J, Hugo D. Tasmanian marine analysis network—Sullivans Cove CSIRO Wharf Sensor. 2012;. https://doi.org/10.4225/08/50613AE767787. https://data.csiro.au/collections/#collection/CIcsiro:5604v1. Accessed 08 May 2019. Wren CR, Ivanov YA, Leigh D, Westhues J. The merl motion detector dataset. In: Workshop on massive datasets (MD). 2007. pp. 10–14. http://www.merl.com/publications/TR2007-069. Wren C, Ivanov Y. MERLSense Data (2009). https://sites.google.com/a/drwren.com/wmd/home. Accessed 08 May 2019. PhysioNet: PhysioNet: the research resource for complex physiologic signals (2019). https://physionet.org/. Accessed 08 May 2019. Kruschke J. Bayesian estimation supersedes the t test. J Exp Psychol Gen. 2012;. https://doi.org/10.1037/a0029146. Salvatier J, V Wiecki T, Fonnesbeck C. Probabilistic programming in python using pymc3. 2016. https://doi.org/10.7287/PEERJ.PREPRINTS.1686V1. Chicco D. Ten quick tips for machine learning in computational biology. BioData Mining. 2017. p. 10. https://doi.org/10.1186/s13040-017-0155-3. Accessed 17 Mar 2019.