Towards emotionally intelligent buildings: A Convolutional neural network based approach to classify human emotional experience in virtual built environments

Ackermann, 2016, EEG-based automatic emotion recognition: Feature extraction, selection and classification methods, 1 Alhagry, 2017, Emotion recognition based on EEG using LSTM recurrent neural network, Emotion, 8, 355 Alhalaseh, 2020, Machine-learning-based emotion recognition system using eeg signals, Computers, 9, 95, 10.3390/computers9040095 Alsolamy, 2016, Emotion estimation from EEG signals during listening to Quran using PSD features, 1 Aryal, 2017, Monitoring fatigue in construction workers using physiological measurements, Autom. Constr., 82, 154, 10.1016/j.autcon.2017.03.003 Aspinall, 2015, The urban brain: analysing outdoor physical activity with mobile EEG, Br. J. Sports Med., 49, 272, 10.1136/bjsports-2012-091877 Atkinson, 2016, Improving BCI-based emotion recognition by combining EEG feature selection and kernel classifiers, Expert Syst. Appl., 47, 35, 10.1016/j.eswa.2015.10.049 Azzazy, 2021, A critical review on the impact of built environment on users’ measured brain activity, Archit. Sci. Rev., 64, 319, 10.1080/00038628.2020.1749980 Bar, 2006, Human prefer curved visual objects, Psychol. Sci., 17, 645, 10.1111/j.1467-9280.2006.01759.x Banaei, 2015, Enhancing urban trails design quality by using electroencephalography Device, Procedia Soc. Behav. Sci., 201, 386, 10.1016/j.sbspro.2015.08.191 Banaei, 2017, Walking through architectural spaces: the impact of interior forms on human brain dynamics, Front. Hum. Neurosci., 11, 477, 10.3389/fnhum.2017.00477 Berto, 2014, The role of nature in coping with psycho-physiological stress: a literature review on restorativeness, Behavioral Sciences, 4, 394, 10.3390/bs4040394 Beukeboom, 2012, Stress-Reducing effects of real and artificial nature in a hospital waiting room, J. Altern. Complement. Med., 18, 329, 10.1089/acm.2011.0488 Bhatti, 2016, Human emotion recognition and analysis in response to audio music using brain signals, Comput. Hum. Behav., 65, 267, 10.1016/j.chb.2016.08.029 Chang, 2008, Psychophysiological responses and restorative values of natural environments in Taiwan, Landsc. Urban Plan., 85, 79, 10.1016/j.landurbplan.2007.09.010 Chen, 2016, Revealing the “invisible gorilla” in construction: Estimating construction safety through mental workload assessment, Autom. Constr., 63, 173, 10.1016/j.autcon.2015.12.018 Chen, 2019, Accurate EEG-based emotion recognition on combined features using deep convolutional neural networks, IEEE Access, 7, 44317, 10.1109/ACCESS.2019.2908285 Cohen, 2017, Where does EEG come from and what does it mean?, Trends Neurosci., 40, 208, 10.1016/j.tins.2017.02.004 Comon, 1994, Independent component analysis, a new concept?, Signal Process., 36, 287, 10.1016/0165-1684(94)90029-9 Delorme, A., & Makeig, S. (2004). “EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis”. Journal of neuroscience methods, 134(1), 9-2. Dravigne, 2008, The effect of live plants and window views of green spaces on employee perceptions of job satisfaction, HortSci., 43, 183, 10.21273/HORTSCI.43.1.183 Dudek, 2000 Du, J., Shi, Y., Zou, Z., and Zhao, D. (2017). “CoVR: Cloud-Based Multiuser Virtual Reality Headset System for Project Communication of Remote Users.” Journal of Construction Engineering and Management, 144(2), 04017109.1- 19. Dwivedi, 2022, Metaverse beyond the hype: Multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy, Int. J. Inf. Manag., 66, 10.1016/j.ijinfomgt.2022.102542 Ergan, S., Shi, Z., & Yu, X. (2018). “Towards Quantifying Human Experience in the Built Environment: A Crowdsourcing Based Experiment to Identify Influential Architectural Design Features”. Journal of Building Engineering. Ergan, S., Radwan, A., Zou, Zhengbo, Tseng, H., and Han, X. (2018). “Quantifying human experience in architectural spaces with integrated virtual reality and body sensor networks.” ASCE Journal of Computing in Civil Engineering, 33(2), 04018062.1- 13. Essawy, 2014, Timeless buildings and the human brain: The effect of spiritual spaces on human brain waves, Archnet-IJAR: International Journal of Architectural Research, 8, 133, 10.26687/archnet-ijar.v8i1.329 Fan, 2017, EEG-based affect and workload recognition in a virtual driving environment for ASD intervention, IEEE Trans. Biomed. Eng., 65, 43, 10.1109/TBME.2017.2693157 Goldhagen, S. W. (2017). How the built environment shapes our lives. Harper Collins Publisher, NY, USA. Gonzalez, H. A., Yoo, J., & Elfadel, I. M. (2019, July). “EEG-based emotion detection using unsupervised transfer learning”. In 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (pp. 694-697). Hadjidimitriou, 2012, Toward an EEG-based recognition of music liking using time-frequency analysis, IEEE Trans. Biomed. Eng., 59, 3498, 10.1109/TBME.2012.2217495 Hamann, 2012, Mapping discrete and dimensional emotions onto the brain: controversies and consensus, Trends Cogn. Sci., 16, 458, 10.1016/j.tics.2012.07.006 Heydarian, 2015, Immersive virtual environments versus physical built environments: A benchmarking study for building design and user-built environment explorations, Autom. Constr., 54, 116, 10.1016/j.autcon.2015.03.020 Hidayetoglu, 2010, The effects of training and spatial experience on the perception of the interior of buildings with a high level of complexity, Sci. Res. Essays, 5, 428 Higuera-Trujillo, J. L., Llinares Millan, C., Montanana i Avino, A., & Rojas, J. C. (2020). Multisensory stress reduction: a neuro-architecture study of paediatric waiting rooms. Building Research & Information, 48(3), 269-285. Hollander, 2016, Brain responses to architecture and planning: A preliminary neuro-assessment of the pedestrian experience in Boston, Massachusetts. Architectural Science Review, 59, 474, 10.1080/00038628.2016.1221499 Hosseini, S. A., & Khalilzadeh, M. A. (2010, April). “Emotional stress recognition system using EEG and psychophysiological signals: Using new labelling process of EEG signals in emotional stress state”. In 2010 international conference on biomedical engineering and computer science (pp. 1-6). Jalilifard, 2016, Emotion classification using single-channel scalp-EEG recording, 845 Jebelli, 2018, EEG-based workers' stress recognition at construction sites, Autom. Constr., 93, 315, 10.1016/j.autcon.2018.05.027 Jebelli, 2019, Application of wearable biosensors to construction sites. I: Assessing workers’ stress, J. Constr. Eng. Manag., 145, 04019079, 10.1061/(ASCE)CO.1943-7862.0001729 Karmann, C., Schiavon, S., & Arens, E. (2018). Percentage of commercial buildings showing at least 80% occupant satisfied with their thermal comfort. Karandinou, 2017, Architecture and neuroscience; what can the EEG recording of brain activity reveal about a walk through everyday spaces?, Int. J. Parallel Emergent Distrib. Syst., 32, S54, 10.1080/17445760.2017.1390089 Khalaj, 2014, Comparison of semantic intent and realization in product design: a study on high-end furniture impressions, Int. J. Des., 8, 79 Kim, 2018, Virtual reality sickness questionnaire (VRSQ): Motion sickness measurement index in a virtual reality environment, Appl. Ergon., 69, 66, 10.1016/j.apergo.2017.12.016 Kim, M. K., Kim, M., Oh, E., & Kim, S. P. (2013). “A review on the computational methods for emotional state estimation from the human EEG”. Computational and mathematical methods in medicine, 2013. Kingma, D. P., & Ba, J. (2014). “Adam: A method for stochastic optimization”. arXiv preprint arXiv:1412.6980. Klepeis, 2001, The National Human Activity Pattern Survey (NHAPS): a resource for assessing exposure to environmental pollutants, J. Eposure Sci. Environ. Epidemiol., 11, 231, 10.1038/sj.jea.7500165 Lee, 2017, A study on immersion and VR sickness in walking interaction for immersive virtual reality applications, Symmetry, 9, 68, 10.3390/sym9050068 Lee, 2018, Temporal effect of construction workforce physical strain on diminishing marginal productivity at the task level, J. Constr. Eng. Manag., 144, 04018083, 10.1061/(ASCE)CO.1943-7862.0001531 Li, M., & Lu, B. L. (2009, September). “Emotion classification based on gamma-band EEG”. In 2009 Annual International Conference of the IEEE Engineering in medicine and biology society (pp. 1223-1226). Li, 2018, Emotion recognition from multichannel EEG signals using K-nearest neighbor classification, Technol. Health Care, 26, 509, 10.3233/THC-174836 Li, 2019, Pre-service fatigue screening for construction workers through wearable EEG-based signal spectral analysis, Autom. Constr., 106, 10.1016/j.autcon.2019.102851 Li, 2020, Building environment information and human perceptual feedback collected through a combined virtual reality (VR) and electroencephalogram (EEG) method, Energ. Buildings, 224, 10.1016/j.enbuild.2020.110259 Lindsley, 1952, Psychological phenomena and the electroencephalogram, Electroencephalogr. Clin. Neurophysiol., 4, 443, 10.1016/0013-4694(52)90075-8 Liu, S., Tong, J., Xu, M., Yang, J., Qi, H., & Ming, D. (2016, August). “Improve the generalization of emotional classifiers across time by using training samples from different days”. In 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (pp. 841-844). Lustig, A., and Lustig M. (2014). “The role of visual attention in architectural design.” Proceedings of 2014 ANFA Conference, Salk Institute for Biological Studies, September 18-20, 2014, La Jolla, CA, 60-61. Mavros, P., Coyne, R., Roe, J., & Aspinall, P. (2012). Engaging the brain: Implications of mobile EEG for spatial representation. In Proceedings of the 30th International Conference on Education and Research in Computer Aided Architectural Design in Europe, Prague, Czech Republic, 12-14 September 2012. Murcia, 2019, Neural representation of different 3D architectural images: An EEG study, Integr. Comput.-Aided Eng., 26, 197, 10.3233/ICA-180591 Nakisa, 2020, Automatic emotion recognition using temporal multimodal deep learning, IEEE Access, 8, 225463, 10.1109/ACCESS.2020.3027026 Niedermeyer, E., & da Silva, F. L. (Eds.). (2005). Electroencephalography: basic principles, clinical applications, and related fields. Lippincott Williams & Wilkins. Panksepp, 2004 Petrantonakis, 2009, Emotion recognition from EEG using higher order crossings, IEEE Trans. Inf Technol. Biomed., 14, 186, 10.1109/TITB.2009.2034649 Picard, 2001, Toward machine emotional intelligence: Analysis of affective physiological state, IEEE Trans. Pattern Anal. Mach. Intell., 23, 1175, 10.1109/34.954607 Posner, 2005, The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology, Dev. Psychopathol., 17, 715, 10.1017/S0954579405050340 Rashid, 2008, A review of the empirical literature on the relationships between indoor environment and stress in health care and office settings: problems and prospects of sharing evidence, Environ. Behav., 40, 151, 10.1177/0013916507311550 Roberts, C. J., Edwards, D. J., Hosseini, M. R., Mateo-Garcia, M., & Owusu-Manu, D. G. (2019). “Post-occupancy evaluation: a review of literature”. Engineering, Construction and Architectural Management. Roy, 2019, Deep learning-based electroencephalography analysis: a systematic review, J. Neural Eng., 16, 10.1088/1741-2552/ab260c Russell, 1980, A circumplex model of affect, J. Pers. Soc. Psychol., 39, 1161, 10.1037/h0077714 Schiff, 1994, Fast wavelet transformation of EEG, Electroencephalogr. Clin. Neurophysiol., 91, 442, 10.1016/0013-4694(94)90165-1 Shemesh, 2017, Affective response to architecture–investigating human reaction to spaces with different geometry, Archit. Sci. Rev., 60, 116, 10.1080/00038628.2016.1266597 Shemesh, 2022, The emotional influence of different geometries in virtual spaces: A neurocognitive examination, J. Environ. Psychol., 81, 10.1016/j.jenvp.2022.101802 Snow, 2021, The treachery of images: how realism influences brain and behavior, Trends Cogn. Sci., 25, 506, 10.1016/j.tics.2021.02.008 Snyder, 1987 Suhaimi, N. S., Mountstephens, J., & Teo, J. (2020). EEG-based emotion recognition: A state-of-the-art review of current trends and opportunities. Computational intelligence and neuroscience, 2020. Sun, 2020, Use of electroencephalography (EEG) for comparing study of the external space perception of traditional and modern commercial districts, Journal of Asian Architecture and Building Engineering, 1 Steinwart, 2008 Sternberg, 2006, Neuroscience and architecture: seeking common ground, Cell, 127, 239, 10.1016/j.cell.2006.10.012 Stone, 1998, Task type, posters, and workspace color on mood, satisfaction, and performance, J. Environ. Psychol., 18, 175, 10.1006/jevp.1998.0084 Tilley, 2017, Older people’s experiences of mobility and mood in an urban environment: A mixed methods approach using electroencephalography (EEG) and interviews, Int. J. Environ. Res. Public Health, 14, 151, 10.3390/ijerph14020151 Tomkins, S. (1962). Affect imagery consciousness: Volume I: The positive affects. Springer publishing company. Ulrich, 2008, Healthcare leadership: a review of the research literature on evidence-based healthcare design, Health Environments Research & Design Journal, 1 Umer, 2020, Physical exertion modeling for construction tasks using combined cardiorespiratory and thermoregulatory measures, Autom. Constr., 112, 10.1016/j.autcon.2020.103079 Vartanian, 2015, Architectural design and the brain: Effects of ceiling height and perceived enclosure on beauty judgments and approach-avoidance decisions, J. Environ. Psychol., 41, 10, 10.1016/j.jenvp.2014.11.006 Van den Berg, 2007, Preference for nature in urbanized societies: stress, restoration, and the pursuit of sustainability, J. Soc. Issues, 63, 79, 10.1111/j.1540-4560.2007.00497.x Vecchiato, 2015, Neurophysiological correlates of embodiment and motivational factors during the perception of virtual architectural environments, Cogn. Process., 16, 425, 10.1007/s10339-015-0725-6 Veitch, J. A., and Galasiu, A. D. (2012). “The physiological and psychological effects of windows, daylight, and view at home review and research agenda no. IRC-RR-325.“ NRC Institute for Research in Construction, Ottawa, Canada. Vorobyov, S., and A. Cichocki. 2002. “Blind noise reduction for multisen- sory signals using ICA and subspace filtering, with application to EEG analysis.” Biol. Cybern. 86 (4): 293–303. https://doi.org/10.1007 /s00422-001-0298-6. Watson, 1988, Positive and negative affectivity and their relation to anxiety and depressive disorders, J. Abnorm. Psychol., 97, 346, 10.1037/0021-843X.97.3.346 Wang, 2019, Detecting and measuring construction workers' vigilance through hybrid kinematic-EEG signals, Autom. Constr., 100, 11, 10.1016/j.autcon.2018.12.018 Wu, 2010, The analysis of emotion recognition from GSR based on PSO, 360 Xing, 2020, Effects of physical fatigue on the induction of mental fatigue of construction workers: A pilot study based on a neurophysiological approach, Autom. Constr., 120, 10.1016/j.autcon.2020.103381 Xu, 2017, Active electrodes for wearable EEG acquisition: Review and electronics design methodology, IEEE Rev. Biomed. Eng., 10, 187, 10.1109/RBME.2017.2656388 Yang, 2017, Collective sensing of workers' gait patterns to identify fall hazards in construction, Autom. Constr., 82, 166, 10.1016/j.autcon.2017.04.010 Yegnanarayana, B. (2009). Artificial neural networks. PHI Learning Pvt. Ltd. Zamanian, 2018, A new feature extraction method to improve emotion detection using EEG signals, ELCVIA: electronic letters on computer vision and image analysis, 17, 29, 10.5565/rev/elcvia.1045 Zhuang, 2018, Investigating patterns for self-induced emotion recognition from EEG signals, Sensors, 18, 841, 10.3390/s18030841 Zou, 2017, Emotional response–based approach for assessing the sense of presence of participants in virtual building evacuation studies, J. Comput. Civ. Eng., 31, 04017028, 10.1061/(ASCE)CP.1943-5487.0000679 Zou, Z., Yu, X., & Ergan, S. (2019). “Integrating biometric sensors, VR, and machine learning to classify EEG signals in alternative architecture designs”. In Computing in civil engineering 2019: Visualization, information modeling, and simulation (pp. 169-176). Reston, VA: American Society of Civil Engineers. Zou, 2021, Evaluating the effectiveness of biometric sensors and their signal features for classifying human experience in virtual environments, Adv. Eng. Inf., 49, 10.1016/j.aei.2021.101358