Sensing beyond itself: Multi-functional use of ubiquitous signals towards wearable applications

Seneviratne, 2017, A survey of wearable devices and challenges, IEEE Commun. Surv. Tutor., 19, 2573, 10.1109/COMST.2017.2731979 Pentland, 2000, Looking at people: sensing for ubiquitous and wearable computing, IEEE Trans. Pattern Anal. Mach. Intell., 22, 107, 10.1109/34.824823 Paulovich, 2018, A future with ubiquitous sensing and intelligent systems, ACS Sens., 3, 1433, 10.1021/acssensors.8b00276 Dargie, 2017 Dix, 2003 Patel, 2020, Wearable inertial sensors to measure gait and posture characteristic differences in older adult fallers and non-fallers: a scoping review, Gait Posture, 76, 110, 10.1016/j.gaitpost.2019.10.039 Bet, 2019, Fall detection and fall risk assessment in older person using wearable sensors: a systematic review, Int. J. Med. Inform., 130, 10.1016/j.ijmedinf.2019.08.006 Benson, 2018, The use of wearable devices for walking and running gait analysis outside of the lab: a systematic review, Gait Posture, 63, 124, 10.1016/j.gaitpost.2018.04.047 Kumari, 2017, Increasing trend of wearables and multimodal interface for human activity monitoring: a review, Biosens. Bioelectron., 90, 298, 10.1016/j.bios.2016.12.001 Athavale, 2017, Biosignal monitoring using wearables: observations and opportunities, Biomed. Signal Process. Control, 38, 22, 10.1016/j.bspc.2017.03.011 Dinh, 2020, Stretchable respiration sensors: advanced designs and multifunctional platforms for wearable physiological monitoring, Biosens. Bioelectron., 166, 10.1016/j.bios.2020.112460 King, 2017, Application of data fusion techniques and technologies for wearable health monitoring, Med. Eng. Phys., 42, 1, 10.1016/j.medengphy.2016.12.011 Can, 2019, Stress detection in daily life scenarios using smart phones and wearable sensors: a survey, J. Biomed. Inform., 92, 10.1016/j.jbi.2019.103139 Lietz, 2019, Survey of mood detection through various input modes, 28 Hong, 2019, Microwave sensing and sleep: noncontact sleep-monitoring technology with microwave biomedical radar, IEEE Microw. Mag., 20, 18, 10.1109/MMM.2019.2915469 Alharbi, 2020, Design and evaluation of an authentication framework for wearable devices, IEEE Access, 8, 80369, 10.1109/ACCESS.2020.2990861 Blasco, 2016, A survey of wearable biometric recognition systems, ACM Comput. Surv., 49, 1, 10.1145/2968215 Raij, 2011, Privacy risks emerging from the adoption of innocuous wearable sensors in the mobile environment, 11 Xue, 2019, A review on intelligent wearables: uses and risks, Hum. Behav. Emerg. Technol., 1, 287, 10.1002/hbe2.173 Li, 2020, Triboelectric nanogenerators enabled Internet of things: a survey, Intell. Converg. Netw., 1, 115, 10.23919/ICN.2020.0008 Hiremath, 2014, Wearable Internet of things: concept, architectural components and promises for person-centered healthcare, 304 Cirani, 2015, Wearable computing for the Internet of things, IT Prof., 17, 35, 10.1109/MITP.2015.89 Wearables - Worldwide market forecast. Williamson, 2015, Data sensing and analysis: challenges for wearables, 136 Judy, 2001, Microelectromechanical systems (MEMS): fabrication, design and applications, Smart Mater. Struct., 10, 1115, 10.1088/0964-1726/10/6/301 Khokhar, 1993, Heterogeneous computing: challenges and opportunities, Computer, 26, 18, 10.1109/2.214439 Liu, 2016, Delay-optimal computation task scheduling for mobile-edge computing systems, 1451 Abbas, 2017, Mobile edge computing: a survey, IEEE Int. Things J., 5, 450, 10.1109/JIOT.2017.2750180 Mao, 2017, A survey on mobile edge computing: the communication perspective, IEEE Commun. Surv. Tutor., 19, 2322, 10.1109/COMST.2017.2745201 Lai, 2018, 1 Al-Turjman, 2019, Machine learning for wearable IoT-based applications: a survey Warden, 2019 Lin, 2020, 11711 xbox™ RealSense™ Leap Motion Adib, 2014, 3D tracking via body radio reflections, 317 Zhao, 2018, RF-based 3D skeletons, 267 Adib, 2015, Multi-person localization via RF body reflections, 279 Qian, 2017, Widar: decimeter-level passive tracking via velocity monitoring with commodity Wi-Fi, 1 Qian, 2018, Widar2. 0: passive human tracking with a single Wi-Fi link, 350 Wu, 2020, MmTrack: passive multi-person localization using commodity millimeter wave radio, IEEE Int. Conf. Comput. Commun. Jovicic, 2013, Visible light communication: opportunities, challenges and the path to market, IEEE Commun. Mag., 51, 26, 10.1109/MCOM.2013.6685754 Arnon, 2015 Song, 2015, An indoor broadband broadcasting system based on PLC and VLC, IEEE Trans. Broadcast., 61, 299, 10.1109/TBC.2015.2400825 Li, 2015, Human sensing using visible light communication, 331 Li, 2016, Practical human sensing in the light, 71 Ibrahim, 2016, Visible light based activity sensing using ceiling photosensors, 43 Yang, 2015, Wearables can afford: light-weight indoor positioning with visible light, 317 Faulkner, 2019, Smart wall: passive visible light positioning with ambient light only, 1 Konings, 2019, FieldLight: device-free indoor human localization using passive visible light positioning and artificial potential fields, IEEE Sens. J., 20, 1054, 10.1109/JSEN.2019.2944178 Lien, 2016, Soli: ubiquitous gesture sensing with millimeter wave radar, ACM Trans. Graph., 35, 1, 10.1145/2897824.2925953 Gong, 2020, Acustico: surface tap detection and localization using wrist-based acoustic TDOA sensing, 406 Chen, 2020, A low latency on-body typing system through single vibration sensor, IEEE Trans. Mob. Comput., 19, 2520, 10.1109/TMC.2019.2928549 Ling, 2018, UltraGesture: fine-grained gesture sensing and recognition, 1 Zhang, 2016, User-friendly fine-grained gesture recognition using WiFi signals, 83 Zheng, 2019, Zero-effort cross-domain gesture recognition with Wi-Fi, 313 Ma, 2018, SignFi: sign language recognition using WiFi, Proc. ACM Interact., Mob., Wearable Ubiquitous Technol., 2, 1, 10.1145/3191755 Han, 2020, In-air handwriting by passive gesture tracking using commodity WiFi, IEEE Commun. Lett., 24, 2652, 10.1109/LCOMM.2020.3007982 Zhang, 2016, Recognizing on-skin input for smartwatches, 13 Zhang, 2015, Soundwrite: text input on surfaces through mobile acoustic sensing, 13 Du, 2018, Accurate acoustic-based handwriting recognition using deep learning, IEEE Int. Conf. Comput. Commun., 1448 Yin, 2019, Ubiquitous writer: robust text input for small mobile devices via acoustic sensing, IEEE Int. Things J., 6, 5285, 10.1109/JIOT.2019.2900355 Zou, 2019, AcouDigits: enabling users to input digits in the air, IEEE Int. Conf. Pervasive Comput. Commun., 1 Wu, 2021, EchoWrite: an acoustic-based finger input system without training, IEEE Trans. Mob. Comput., 20, 1789, 10.1109/TMC.2020.2973094 Wang, 2016, Device-free gesture tracking using acoustic signals, 82 Chen, 2018, WritePad: consecutive number writing on your hand with smart acoustic sensing, IEEE Access, 6, 77240, 10.1109/ACCESS.2018.2880980 Lu, 2020, Enable traditional laptops with virtual writing capability leveraging acoustic signals, Comput. J. Zhang, 2017, FingerSound: recognizing unistroke thumb gestures using a ring, Proc. ACM Interact., Mob., Wearable Ubiquitous Technol., 1, 10.1145/3130985 Cao, 2018, IPand: accurate gesture input with ambient acoustic sensing on hand, IEEE Int. Performance Comput. Commun Conf., 1 Luo, 2020, HCI on the table: robust gesture recognition using acoustic sensing in your hand, IEEE Access, 8, 31481, 10.1109/ACCESS.2020.2973305 Zhang, 2017, SoundTrak, Proc. ACM Interact., Mob., Wearable Ubiquitous Technol., 1, 1 Mao, 2016, CAT: high-precision acoustic motion tracking, 69 Wang, 2019, MilliSonic: Pushing the Limits of Acoustic Motion Tracking, 10.1145/3290605.3300248 Li, 2020, FM-track: pushing the limits of contactless multi-target tracking using acoustic signals, 150 Yun, 2017, Strata: fine-grained acoustic-based device-free tracking, 15 Liu, 2019, Accurate and fast acoustic motion tracking using mobile devices, IEEE Trans. Mob. Comput. Cao, 2020, EarphoneTrack: involving earphones into the ecosystem of acoustic motion tracking, 95 Lee, 2020, Mechano-acoustic sensing of physiological processes and body motions via a soft wireless device placed at the suprasternal notch, Nat. Biomed. Eng., 4, 148, 10.1038/s41551-019-0480-6 Luguern, 2021, Wavelet variance maximization: a contactless respiration rate estimation method based on remote photoplethysmography, Biomed. Signal Process. Control, 63, 10.1016/j.bspc.2020.102263 Wang, 2018, Rf-ecg: heart rate variability assessment based on cots rfid tag array, Proc. ACM Interact., Mob., Wearable Ubiquitous Technol., 2, 1 Wang, 2020, Resilient respiration rate monitoring with realtime bimodal CSI data, IEEE Sens. J. Gao, 2020, Device-free multi-person respiration monitoring using WiFi, IEEE Trans. Veh. Technol., 10.1109/TVT.2020.3020180 Wang, 2020, On CSI-based vital sign monitoring using commodity WiFi, ACM Trans. Comput. Healthc., 1, 1 Wang, 2020, Vital sign monitoring using commodity millimeter wave radio, 8304 Adib, 2015, Smart homes that monitor breathing and heart rate, 837 Yue, 2018, Extracting multi-person respiration from entangled RF signals, 1 Zhao, 2020, Heart rate sensing with a robot mounted mmWave radar, 2812 Ha, 2020, Contactless Seismocardiography via Deep Learning Radars, 10.1145/3372224.3419982 Xie, 2021, Non-contact respiration detection leveraging music and broadcast signals, IEEE Int. Things J., 8, 2931, 10.1109/JIOT.2020.3021915 Xu, 2020, Leveraging acoustic signals for fine-grained breathing monitoring in driving environments, IEEE Trans. Mob. Comput. Song, 2020, SpiroSonic: monitoring human lung function via acoustic sensing on commodity smartphones, 1 Qian, 2018, Acousticcardiogram: monitoring heartbeats using acoustic signals on smart devices, IEEE Int. Conf. Comput. Commun., 1574 Wang, 2018, Monitoring of the central blood pressure waveform via a conformal ultrasonic device, Nat. Biomed. Eng., 2, 687, 10.1038/s41551-018-0287-x Sharma, 2019, Acoustic sensing as a novel wearable approach for cardiac monitoring at the wrist, Sci. Rep., 9, 1, 10.1038/s41598-019-55599-5 Cotur, 2020, Stretchable composite acoustic transducer for wearable monitoring of vital signs, Adv. Funct. Mater., 30 Liu, 2016, Epidermal mechano-acoustic sensing electronics for cardiovascular diagnostics and human-machine interfaces, Sci. Adv., 2, 10.1126/sciadv.1601185 Artemyev, 2020, Robust algorithm for remote photoplethysmography in realistic conditions, Digit. Signal Process., 104, 10.1016/j.dsp.2020.102737 Macwan, 2018, Periodic variance maximization using generalized eigenvalue decomposition applied to remote photoplethysmography estimation, 1332 Rouast, 2018, Remote heart rate measurement using low-cost RGB face video: a technical literature review, Front. Comput. Sci., 12, 858, 10.1007/s11704-016-6243-6 Blackford, 2016, Long-range non-contact imaging photoplethysmography: cardiac pulse wave sensing at a distance, optical diagnostics and sensing XVI: toward point-of-care diagnostics, Int. Soc. Opt. Photonics Luguern, 2020, Remote photoplethysmography combining color channels with SNR maximization for respiratory rate assessment, 1 Huang, 2020, A heart rate monitoring framework for real-world drivers using remote photoplethysmography, IEEE J. Biomed. Health Inform. Tsou, 2020, Multi-task learning for simultaneous video generation and remote photoplethysmography estimation Botina-Monsalve, 2020, Long short-term memory deep-filter, 306 Tsou, 2020, Siamese-rPPG network: remote photoplethysmography signal estimation from face videos, 2066 Song, 2021, PulseGAN: learning to generate realistic pulse waveforms in remote photoplethysmography, IEEE J. Biomed. Health Inform., 25, 1373, 10.1109/JBHI.2021.3051176 Wang, 2018, Single element remote-PPG, IEEE Trans. Biomed. Eng. Liu, 2020, Wi-PSG: detecting rhythmic movement disorder using COTS WiFi, IEEE Int. Things J. Clemente, 2019, Indoor person identification and fall detection through non-intrusive floor seismic sensing, 417 Clemente, 2020, Real-time contact-free monitoring of sleep activities and events around the bed, IEEE Int. Conf. Pervasive Comput. Commun., 1 Zhou, 2020, Single-layered ultra-soft washable smart textiles for all-around ballistocardiograph, respiration, and posture monitoring during sleep, Biosens. Bioelectron., 155, 10.1016/j.bios.2020.112064 Liu, 2018, Monitoring vital signs and postures during sleep using WiFi signals, IEEE Int. Things J., 5, 2071, 10.1109/JIOT.2018.2822818 Zhang, 2019, Smars: sleep monitoring via ambient radio signals, IEEE Trans. Mob. Comput. Yue, 2020, BodyCompass, Proc. ACM Interact., Mob., Wearable Ubiquitous Technol., 4, 1, 10.1145/3397311 Hsu, 2017, Zero-effort in-home sleep and insomnia monitoring using radio signals, Proc. ACM Interact., Mob., Wearable Ubiquitous Technol., 1, 1, 10.1145/3130924 Liu, 2019, Tagsheet: sleeping posture recognition with an unobtrusive passive tag matrix, IEEE Int. Conf. Comput. Commun., 874 Tian, 2018, RF-based fall monitoring using convolutional neural networks Wang, 2017, WiFall: device-free fall detection by wireless networks, IEEE Trans. Mob. Comput., 16, 581, 10.1109/TMC.2016.2557792 Ding, 2020, A WiFi-based smart home fall detection system using recurrent neural network, IEEE Trans. Consum. Electron., 66, 308, 10.1109/TCE.2020.3021398 Li, 2018, Non-intrusive and non-contact sleep monitoring with seismometer, 449 Celik, 2021, Gait analysis in neurological populations: progression in the use of wearables, Med. Eng. Phys., 87, 9, 10.1016/j.medengphy.2020.11.005 Greene, 2015, Assessment and classification of early-stage multiple sclerosis with inertial sensors: comparison against clinical measures of disease state, IEEE J. Biomed. Health Inform., 19, 1356, 10.1109/JBHI.2015.2435057 Chang, 2016, A wearable inertial measurement system with complementary filter for gait analysis of patients with stroke or Parkinson's disease, IEEE Access, 4, 8442, 10.1109/ACCESS.2016.2633304 Yoon, 2020, Neural correlates and gait characteristics for hypoxic-ischemic brain injury induced freezing of gait, Clin. Neurophysiol., 131, 46, 10.1016/j.clinph.2019.09.027 Prince, 2007, No health without mental health, Lancet, 370, 859, 10.1016/S0140-6736(07)61238-0 Zvarevashe, 2020, Ensemble learning of hybrid acoustic features for speech emotion recognition, Algorithms, 13 Kim, 2020, Affective latent representation of acoustic and lexical features for emotion recognition, Sensors, 20 Gravina, 2019, Emotion-relevant activity recognition based on smart cushion using multi-sensor fusion, Inf. Fusion, 48, 1, 10.1016/j.inffus.2018.08.001 Hashmi, 2020, Motion reveal emotions: identifying emotions from human walk using chest mounted smartphone, IEEE Sens. J., 20, 13511, 10.1109/JSEN.2020.3004399 Zhu, 2017, Emotion recognition from Chinese speech for smart affective services using a combination of SVM and DBN, Sensors, 17, 10.3390/s17071694 Mosciano, 2017, An array of physical sensors and an adaptive regression strategy for emotion recognition in a noisy scenario, Sens. Actuators A, Phys., 267, 48, 10.1016/j.sna.2017.09.056 Kaya, 2018, Efficient and effective strategies for cross-corpus acoustic emotion recognition, Neurocomputing, 275, 1028, 10.1016/j.neucom.2017.09.049 Chen, 2018, Transfer learning for wearable long-term social speech evaluations, IEEE Access, 6, 61305, 10.1109/ACCESS.2018.2876122 Yang, 2019, IoT structured long-term wearable social sensing for mental wellbeing, IEEE Int. Things J., 6, 3652, 10.1109/JIOT.2018.2889966 Gu, 2017, Wearable social sensing: content-based processing methodology and implementation, IEEE Sens. J., 17, 7167, 10.1109/JSEN.2017.2754289 Jin, 2020, Attention-block deep learning based features fusion in wearable social sensor for mental wellbeing evaluations, IEEE Access, 8, 89258, 10.1109/ACCESS.2020.2994124 Arpaia, 2020, A wearable EEG instrument for real-time frontal asymmetry monitoring in worker stress analysis, IEEE Trans. Instrum. Meas., 69, 8335, 10.1109/TIM.2020.2988744 Coutts, 2020, Deep learning with wearable based heart rate variability for prediction of mental and general health, J. Biomed. Inform., 112, 10.1016/j.jbi.2020.103610 Zanetti, 2019, Multilevel assessment of mental stress via network physiology paradigm using consumer wearable devices, J. Ambient Intell. Humaniz. Comput. Setz, 2010, Discriminating stress from cognitive load using a wearable EDA device, IEEE Trans. Inf. Technol. Biomed., 14, 410, 10.1109/TITB.2009.2036164 Ghandeharioun, 2017, Objective assessment of depressive symptoms with machine learning and wearable sensors data, 325 Zhao, 2016, Emotion recognition using wireless signals, 95 Matkovič, 2018, Wireless mental effort inference, 1241 Zheng, 2020, V2iFi: in-vehicle vital sign monitoring via compact RF sensing, Proc. ACM Interact., Mob., Wearable Ubiquitous Technol., 4, 1, 10.1145/3397321 Liu, 2020, Continuous user verification via respiratory biometrics, IEEE International Conference on Computer Communications, IEEE, 1 Lin, 2017, Cardiac scan: a non-contact and continuous heart-based user authentication system, 315 Li, 2020, VocalPrint: exploring a resilient and secure voice authentication via mmWave biometric interrogation, 312 Kong, 2020, Continuous authentication through finger gesture interaction for smart homes using WiFi, IEEE Trans. Mob. Comput. Arra, 2019, Personalized gait-based authentication using UWB wearable devices, 206 Yang, 2020, Multi-user identification through gaits using millimeter wave radios, IEEE International Conference on Computer Communications, IEEE, 2589 Zhang, 2016, Wifi-id: human identification using WiFi signal, 75 Gao, 2019, EarEcho: using ear canal echo for wearable authentication, Proc. ACM Interact., Mob., Wearable Ubiquitous Technol., 3, 1 Zou, 2018, BiLock: user authentication via dental occlusion biometrics, Proc. ACM Interact., Mob., Wearable Ubiquitous Technol., 2, 1, 10.1145/3264962 Chen, 2020, 1 Shi, 2020, 829 Chauhan, 2017, BreathPrint: breathing acoustics-based user authentication, 278 Zhao, 2020, Continuous Authentication on Wrist-Worn Wearables Using PPG-Based Biometrics, 10.1109/INFOCOM41043.2020.9155526 Zhao, 2019, Toward Continuous User Authentication Using PPG in Commodity Wrist-Worn Wearables, 10.1145/3300061.3343375 Cao, 2020, 1917 Bastos, 2020, Double authentication model based on PPG and ECG signals, Int. Wirel. Commun. Mob. Comput., 601 Karimian, 2017, Human recognition from photoplethysmography (PPG) based on non-fiducial features, 4636 Khan, 2020, 1 Lim, 2019, Deep multiview heartwave authentication, IEEE Trans. Ind. Inform., 15, 777, 10.1109/TII.2018.2874477 Lim, 2019, Heartrate-dependent heartwave biometric identification with thresholding-based GMM–HMM methodology, IEEE Trans. Ind. Inform., 15, 45, 10.1109/TII.2018.2874462 Liu, 2018, Privacy leakage through context-free wearable side-channel, IEEE International Conference on Computer Communications, IEEE, 1232 Li, 2017, ClickLeak: keystroke leaks through multimodal sensors in cyber-physical social networks, IEEE Access, 5, 27311, 10.1109/ACCESS.2017.2776527 Wang, 2018, Personal PIN leakage from wearable devices, IEEE Trans. Mob. Comput., 17, 646, 10.1109/TMC.2017.2737533 Michalevsky, 2014, Gyrophone: recognizing speech from gyroscope signals, 1053 Wang, 2015, Motion leaks through smartwatch sensors, 155 Wang, 2019, WristSpy: snooping passcodes in mobile payment using wrist-worn wearables, IEEE International Conference on Computer Communications, IEEE, 2071 Jiang, 2019, Motion eavesdropper: smartwatch-based handwriting recognition using deep learning, 145 Asonov, 2004, Keyboard acoustic emanations, IEEE Sympos. Secur. Priv., 3 Zhuang, 2005, Keyboard Acoustic Emanations Revisited, 10.1145/1102120.1102169 Berger, 2006, Dictionary attacks using keyboard acoustic emanations, 245 Halevi, 2014, Keyboard acoustic side channel attacks: exploring realistic and security-sensitive scenarios, Int. J. Inf. Secur., 14, 443, 10.1007/s10207-014-0264-7 Yu, 2019, An indirect eavesdropping attack of keystrokes on touch screen through acoustic sensing, IEEE Trans. Mob. Comput. Liu, 2015, Snooping keystrokes with mm-level audio ranging on a single phone, 142 Xiao, 2020, Mobile Phones Know Your Keystrokes Through the Sounds from Finger's Tapping on the Screen, 10.1109/ICDCS47774.2020.00102 Yu, 2016, Writinghacker: audio based eavesdropping of handwriting via mobile devices, 463 Ding, 2019, Human–machine interfacing enabled by triboelectric nanogenerators and tribotronics, Adv. Mater. Technol., 4, 10.1002/admt.201800487 Askari, 2018, Piezoelectric and triboelectric nanogenerators: trends and impacts, Nano Today, 22, 10, 10.1016/j.nantod.2018.08.001 Wen, 2016, Self-powered textile for wearable electronics by hybridizing fiber-shaped nanogenerators, solar cells, and supercapacitors, Sci. Adv., 2, 10.1126/sciadv.1600097 Fan, 2020, Analog sensing and computing systems with low power consumption for gesture recognition, Adv. Intell. Syst., 3 Liu, 2020, A current-mode computation-in-memory architecture enabling near-sensor processing for intelligent IoT vision nodes, IEEE Trans. Circuits Syst. I, Regul. Pap., 67, 2909, 10.1109/TCSI.2020.2984161 Wu, 2019, Deep compressed sensing, 6850 Predd, 2006, Distributed learning in wireless sensor networks, IEEE Signal Process. Mag., 23, 56, 10.1109/MSP.2006.1657817 Pan, 2009, A survey on transfer learning, IEEE Trans. Knowl. Data Eng., 22, 1345, 10.1109/TKDE.2009.191 Yang, 2019, Federated learning, Synth. Lect. Artif. Intell. Mach. Learn., 13, 1