Nội dung được dịch bởi AI, chỉ mang tính chất tham khảo
Học máy và EEG có thể phân loại việc xem thụ động các loại kích thích hình ảnh riêng biệt nhưng không thể phân biệt sự quan sát cơn đau
Tóm tắt
Các nghiên cứu trước đây đã chứng minh tiềm năng của học máy (ML) trong việc phân loại đau thể xác từ các trạng thái không đau bằng cách sử dụng dữ liệu điện não đồ (EEG). Tuy nhiên, ứng dụng của ML đối với dữ liệu EEG để phân loại việc quan sát các hình ảnh thể hiện cơn đau so với những hình ảnh không đau về biểu cảm khuôn mặt con người hoặc các cảnh ngụ ý cơn đau chưa được khám phá. Nghiên cứu hiện tại nhằm giải quyết vấn đề này bằng cách huấn luyện các mô hình Rừng Ngẫu Nhiên (RF) trên các tiềm năng liên quan đến sự kiện (ERP) ghi lại trong khi các tham gia viên ngắm nhìn thụ động những khuôn mặt thể hiện biểu cảm đau hoặc trung hòa, cũng như các cảnh hành động thể hiện cơn đau hoặc những tình huống không đau (trung hòa) tương ứng. Chín mươi mốt người tham gia đã được tuyển chọn từ ba mẫu, bao gồm một nhóm phát triển mô hình (n = 40) và một nhóm xác thực giữa các chủ thể (n = 51). Thêm vào đó, 25 người tham gia từ nhóm phát triển mô hình đã hoàn thành một phiên thí nghiệm thứ hai, cung cấp một mẫu xác thực tạm thời trong chủ thể. Phân tích các ERP cho thấy có sự gia tăng trong thành phần N170 khi so sánh với các cảnh hành động. Hơn nữa, một tiềm năng tích cực muộn (LPP) được quan sát thấy gia tăng trong quá trình xem các cảnh đau so với các cảnh trung hòa. Ngoài ra, một phản ứng P3 được phát hiện gia tăng khi các tham gia viên xem các khuôn mặt thể hiện biểu cảm đau so với các biểu cảm trung hòa. Sau đó, ba mô hình RF được phát triển để phân loại hình ảnh thành các khuôn mặt và cảnh, cảnh trung hòa và đau, và biểu cảm trung hòa và đau. Mô hình RF đạt độ chính xác phân loại lần lượt là 75%, 64%, và 69% cho phân loại xác thực chéo, xác thực giữa các chủ thể và phân loại trong cùng một chủ thể, cùng với các dự đoán được điều chỉnh hợp lý cho phân loại hình ảnh khuôn mặt so với cảnh. Tuy nhiên, mô hình RF không thể phân loại cơn đau so với các kích thích trung hòa với độ chính xác vượt quá xác suất ngẫu nhiên khi được trình bày với các nhiệm vụ tiếp theo liên quan đến hình ảnh từ bất kỳ loại nào. Những kết quả này mở rộng những phát hiện trước đó bằng cách xác thực bên ngoài việc sử dụng ML trong phân loại các ERP liên quan đến các loại hình ảnh trực quan khác nhau, cụ thể là khuôn mặt và cảnh. Các kết quả cũng chỉ ra những hạn chế của ML trong việc phân biệt ý nghĩa đau và không đau bằng cách sử dụng phản ứng ERP đối với việc xem thụ động những hình ảnh tương tự về mặt thị giác.
Từ khóa
#học máy #điện não đồ #phân loại cơn đau #biểu cảm khuôn mặt #tiềm năng liên quan đến sự kiệnTài liệu tham khảo
Stewart AX, Nuthmann A, Sanguinetti G. Single-trial classification of EEG in a visual object task using ICA and machine learning. J Neurosci Methods. 2014;228:1–14. https://doi.org/10.1016/j.jneumeth.2014.02.014.
Zheng X, Chen W, You Y, Jiang Y, Li M, Zhang T. Ensemble deep learning for automated visual classification using EEG signals. Pattern Recognit. 2020;102:107147. https://doi.org/10.1016/j.patcog.2019.107147.
Cudlenco N, Popescu N, Leordeanu M. Reading into the mind’s eye: Boosting automatic visual recognition with EEG signals. Neurocomputing. 2020;386:281–92. https://doi.org/10.1016/j.neucom.2019.12.076.
Bagchi S, Bathula DR. EEG-ConvTransformer for single-trial EEG-based visual stimulus classification. Pattern Recognit. 2022;129:108757. https://doi.org/10.1016/j.patcog.2022.108757.
Yavandhasani M, Ghaderi F. Visual object recognition from single-trial EEG signals using machine learning wrapper techniques. IEEE Trans Biomed Eng. 2022;69(7):2176–83. https://doi.org/10.1109/TBME.2021.3138157.
Kaneshiro B, Perreau Guimaraes M, Kim H-S, Norcia AM, Suppes P. A Representational similarity analysis of the dynamics of object processing using single-trial EEG classification. Najbauer J, ed. PLoS ONE. 2015;10(8):e0135697. https://doi.org/10.1371/journal.pone.0135697.
Ghosh L, Dewan D, Chowdhury A, Konar A. Exploration of face-perceptual ability by EEG induced deep learning algorithm. Biomed Signal Process Control. 2021;66:102368. https://doi.org/10.1016/j.bspc.2020.102368.
Mari T, Henderson J, Maden M, Nevitt S, Duarte R, Fallon N. Systematic review of the effectiveness of machine learning algorithms for classifying pain intensity, phenotype or treatment outcomes using electroencephalogram data. J Pain. 2022;23(3):349–69. https://doi.org/10.1016/j.jpain.2021.07.011.
Mari T, Asgard O, Henderson J, et al. External validation of binary machine learning models for pain intensity perception classification from EEG in healthy individuals. Sci Rep. 2023;13(1):242. https://doi.org/10.1038/s41598-022-27298-1.
van der Miesen MM, Lindquist MA, Wager TD. Neuroimaging-based biomarkers for pain. PAIN Reports. 2019;4(4):e751. https://doi.org/10.1097/PR9.0000000000000751.
Jaworska N, de la Salle S, Ibrahim M-H, Blier P, Knott V. Leveraging machine learning approaches for predicting antidepressant treatment response using electroencephalography (EEG) and clinical data. Front Psychiatry. 2019. https://doi.org/10.3389/fpsyt.2018.00768.
Gram M, Erlenwein J, Petzke F, et al. Prediction of postoperative opioid analgesia using clinical-experimental parameters and electroencephalography. Eur J Pain (United Kingdom). 2017;21(2):264–77. https://doi.org/10.1002/ejp.921.
Graversen C, Olesen SS, Olesen AE, et al. The analgesic effect of pregabalin in patients with chronic pain is reflected by changes in pharmaco-EEG spectral indices. Br J Clin Pharmacol. 2012;73(3):363–72. https://doi.org/10.1111/j.1365-2125.2011.04104.x.
Singer T, Seymour B, O’Doherty J, Kaube H, Dolan RJ, Frith CD. Empathy for pain involves the affective but not sensory components of pain. Science (80-). 2004;303(5661):1157–62. https://doi.org/10.1126/science.1093535.
Decety J, Jackson PL. The Functional Architecture of Human Empathy. Behav Cogn Neurosci Rev. 2004;3(2):71–100. https://doi.org/10.1177/1534582304267187.
Lamm C, Decety J, Singer T. Meta-analytic evidence for common and distinct neural networks associated with directly experienced pain and empathy for pain. Neuroimage. 2011;54(3):2492–502. https://doi.org/10.1016/j.neuroimage.2010.10.014.
Singer T, Lamm C. The social neuroscience of empathy. Ann N Y Acad Sci. 2009;1156(1):81–96. https://doi.org/10.1111/j.1749-6632.2009.04418.x.
Fan Y-T, Chen C, Chen S-C, Decety J, Cheng Y. Empathic arousal and social understanding in individuals with autism: evidence from fMRI and ERP measurements. Soc Cogn Affect Neurosci. 2014;9(8):1203–13. https://doi.org/10.1093/scan/nst101.
Oberman LM, Hubbard EM, McCleery JP, Altschuler EL, Ramachandran VS, Pineda JA. EEG evidence for mirror neuron dysfunction in autism spectrum disorders. Cogn Brain Res. 2005;24(2):190–8. https://doi.org/10.1016/j.cogbrainres.2005.01.014.
Decety J, Holvoet C. The emergence of empathy: a developmental neuroscience perspective. Dev Rev. 2021;62:100999. https://doi.org/10.1016/j.dr.2021.100999.
Preusche I, Lamm C. Reflections on empathy in medical education: what can we learn from social neurosciences? Adv Heal Sci Educ. 2016;21(1):235–49. https://doi.org/10.1007/s10459-015-9581-5.
Coll M-P. Meta-analysis of ERP investigations of pain empathy underlines methodological issues in ERP research. Soc Cogn Affect Neurosci. 2018;13(10):1003–17. https://doi.org/10.1093/scan/nsy072.
Fallon N, Li X, Chiu Y, Nurmikko T, Stancak A. Altered cortical processing of observed pain in patients with fibromyalgia syndrome. J Pain. 2015;16(8):717–26. https://doi.org/10.1016/j.jpain.2015.04.008.
Bötzel K, Schulze S, Stodieck SRG. Scalp topography and analysis of intracranial sources of face-evoked potentials. Exp Brain Res. 1995. https://doi.org/10.1007/BF00229863.
Bentin S, Allison T, Puce A, Perez E, McCarthy G. Electrophysiological studies of face perception in humans. J Cogn Neurosci. 1996;8(6):551–65. https://doi.org/10.1162/jocn.1996.8.6.551.
Itier RJ. N170 or N1? spatiotemporal differences between object and face processing using ERPs. Cereb Cortex. 2004;14(2):132–42. https://doi.org/10.1093/cercor/bhg111.
Eimer M. Effects of face inversion on the structural encoding and recognition of faces. Cogn Brain Res. 2000;10(1–2):145–58. https://doi.org/10.1016/S0926-6410(00)00038-0.
Johnston P, Molyneux R, Young AW. The N170 observed ‘in the wild’: robust event-related potentials to faces in cluttered dynamic visual scenes. Soc Cogn Affect Neurosci. 2015;10(7):938–44. https://doi.org/10.1093/scan/nsu136.
Itier RJ, Taylor MJ. Source analysis of the N170 to faces and objects. NeuroReport. 2004;15(8):1261–5. https://doi.org/10.1097/01.wnr.0000127827.73576.d8.
Soto V, Tyson-Carr J, Kokmotou K, et al. Brain responses to emotional faces in natural settings: a wireless mobile EEG recording study. Front Psychol. 2018. https://doi.org/10.3389/fpsyg.2018.02003.
Jeffreys DA. Evoked potential studies of face and object processing. Vis cogn. 1996;3(1):1–38. https://doi.org/10.1080/713756729.
Jeffreys DA. A face-responsive potential recorded from the human scalp. Exp Brain Res. 1989. https://doi.org/10.1007/BF00230699.
Joyce C, Rossion B. The face-sensitive N170 and VPP components manifest the same brain processes: the effect of reference electrode site. Clin Neurophysiol. 2005;116(11):2613–31. https://doi.org/10.1016/j.clinph.2005.07.005.
Itier RJ, Taylor MJ. Inversion and contrast polarity reversal affect both encoding and recognition processes of unfamiliar faces: a repetition study using ERPs. Neuroimage. 2002;15(2):353–72. https://doi.org/10.1006/nimg.2001.0982.
Varma S, Simon R. Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics. 2006;7(1):91. https://doi.org/10.1186/1471-2105-7-91.
Vabalas A, Gowen E, Poliakoff E, Casson AJ. Machine learning algorithm validation with a limited sample size, Hernandez-Lemus E ed. PLoS ONE. 2019;14(11):e0224365. https://doi.org/10.1371/journal.pone.0224365.
Cabitza F, Campagner A, Soares F, et al. The importance of being external methodological insights for the external validation of machine learning models in medicine. Comput Methods Programs Biomed. 2021;208:106288. https://doi.org/10.1016/j.cmpb.2021.106288.
Lever J, Krzywinski M, Altman N. Model selection and overfitting. Nat Methods. 2016;13(9):703–4. https://doi.org/10.1038/nmeth.3968.
Collins GS, Reitsma JB, Altman DG, Moons K. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement. BMC Med. 2015;13(1):1. https://doi.org/10.1186/s12916-014-0241-z.
Li X, Zhang S, Zhang Q, et al. Diagnosis of thyroid cancer using deep convolutional neural network models applied to sonographic images: a retrospective, multicohort, diagnostic study. Lancet Oncol. 2019;20(2):193–201. https://doi.org/10.1016/S1470-2045(18)30762-9.
Siontis GCM, Tzoulaki I, Castaldi PJ, Ioannidis JPA. External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination. J Clin Epidemiol. 2015;68(1):25–34. https://doi.org/10.1016/j.jclinepi.2014.09.007.
Snell KIE, Archer L, Ensor J, et al. External validation of clinical prediction models: simulation-based sample size calculations were more reliable than rules-of-thumb. J Clin Epidemiol. 2021;135:79–89. https://doi.org/10.1016/j.jclinepi.2021.02.011.
Li A, Wolfe JM, Chen Z. Implicitly and explicitly encoded features can guide attention in free viewing. J Vis. 2020;20(6):8. https://doi.org/10.1167/jov.20.6.8.
Mende-Siedlecki P, Qu-Lee J, Lin J, Drain A, Goharzad A. The Delaware Pain Database: a set of painful expressions and corresponding norming data. PAIN Reports. 2020;5(6):e853. https://doi.org/10.1097/PR9.0000000000000853.
Office for National Statistics (ONS). Ethnic Group, England and Wales: Census 2021.
Fan Y, Han S. Temporal dynamic of neural mechanisms involved in empathy for pain: an event-related brain potential study. Neuropsychologia. 2008;46(1):160–73. https://doi.org/10.1016/j.neuropsychologia.2007.07.023.
Akitsuki Y, Decety J. Social context and perceived agency affects empathy for pain: an event-related fMRI investigation. Neuroimage. 2009;47(2):722–34. https://doi.org/10.1016/j.neuroimage.2009.04.091.
Han S, Fan Y, Mao L. Gender difference in empathy for pain: an electrophysiological investigation. Brain Res. 2008;1196:85–93. https://doi.org/10.1016/j.brainres.2007.12.062.
Fallon N, Li X, Stancak A. Pain catastrophising affects cortical responses to viewing pain in others, Ptito M, ed. PLoS ONE. 2015;10(7):e0133504. https://doi.org/10.1371/journal.pone.0133504.
Sullivan MJL, Bishop SR, Pivik J. The pain catastrophizing scale: development and validation. Psychol Assess. 1995;7(4):524–32. https://doi.org/10.1037/1040-3590.7.4.524.
Gabard-Durnam LJ, Mendez Leal AS, Wilkinson CL, Levin AR. The Harvard automated processing pipeline for electroencephalography (HAPPE): standardized processing software for developmental and high-artifact data. Front Neurosci. 2018. https://doi.org/10.3389/fnins.2018.00097.
Lehmann D. Principles of spatial analysis. In: Gevins AS, Remond A, editors. Handbook of electroencephalography and clinical neurophysiology: methods of analysis of brain electrical and magnetic signals. Amsterdam: Elsevier; 1987. p. 309–54.
Delorme A, Makeig S. EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J Neurosci Methods. 2004;134(1):9–21. https://doi.org/10.1016/j.jneumeth.2003.10.009.
Luo W, Phung D, Tran T, et al. Guidelines for developing and reporting machine learning predictive models in biomedical research: a multidisciplinary view. J Med Internet Res. 2016;18(12):e323. https://doi.org/10.2196/jmir.5870.
Anuragi A, Sisodia DS. Empirical wavelet transform based automated alcoholism detecting using EEG signal features. Biomed Signal Process Control. 2020;57:101777. https://doi.org/10.1016/j.bspc.2019.101777.
Vargas-Lopez O, Perez-Ramirez CA, Valtierra-Rodriguez M, Yanez-Borjas JJ, Amezquita-Sanchez JP. An explainable machine learning approach based on statistical indexes and SVM for stress detection in automobile drivers using electromyographic signals. Sensors. 2021;21(9):3155. https://doi.org/10.3390/s21093155.
Vimala V, Ramar K, Ettappan M. An intelligent sleep Apnea classification system based on EEG signals. J Med Syst. 2019;43(2):36. https://doi.org/10.1007/s10916-018-1146-8.
Sai CY, Mokhtar N, Yip HW, et al. Objective identification of pain due to uterine contraction during the first stage of labour using continuous EEG signals and SVM. Sādhanā. 2019;44(4):87. https://doi.org/10.1007/s12046-019-1058-4.
Kaplan AY, Fingelkurts AA, Fingelkurts AA, Borisov SV, Darkhovsky BS. Nonstationary nature of the brain activity as revealed by EEG/MEG: methodological, practical and conceptual challenges. Signal Process. 2005;85(11):2190–212. https://doi.org/10.1016/j.sigpro.2005.07.010.
Faisal AA, Selen LPJ, Wolpert DM. Noise in the nervous system. Nat Rev Neurosci. 2008;9(4):292–303. https://doi.org/10.1038/nrn2258.
Marathe AR, Ries AJ, McDowell K. Sliding HDCA: single-trial EEG classification to overcome and quantify temporal variability. IEEE Trans Neural Syst Rehabil Eng. 2014;22(2):201–11. https://doi.org/10.1109/TNSRE.2014.2304884.
Maniruzzaman M, Rahman MJ, Al-MehediHasan M, et al. Accurate diabetes risk stratification using machine learning: role of missing value and outliers. J Med Syst. 2018;42(5):92. https://doi.org/10.1007/s10916-018-0940-7.
Pedregosa F, Varoquaux G, Gramfort A, et al. Scikit-learn: machine learning in Python. J Mach Learn Res. 2011;12:2825–30.
Bergstra J, Bengio Y. Random search for hyper-parameter optimization. J Mach Learn Res. 2012;13(2):281–305.
Géron A. Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent, 2nd ed. O’Reilly; 2019.
Yang L, Shami A. On hyperparameter optimization of machine learning algorithms: theory and practice. Neurocomputing. 2020;415:295–316. https://doi.org/10.1016/j.neucom.2020.07.061.
Sokolova M, Lapalme G. A systematic analysis of performance measures for classification tasks. Inf Process Manag. 2009;45(4):427–37. https://doi.org/10.1016/j.ipm.2009.03.002.
Assel M, Sjoberg DD, Vickers AJ. The Brier score does not evaluate the clinical utility of diagnostic tests or prediction models. Diagnostic Progn Res. 2017;1(1):19. https://doi.org/10.1186/s41512-017-0020-3.
Alba AC, Agoritsas T, Walsh M, et al. Discrimination and calibration of clinical prediction models. JAMA. 2017;318(14):1377. https://doi.org/10.1001/jama.2017.12126.
Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW. Calibration: the Achilles heel of predictive analytics. BMC Med. 2019;17(1):230. https://doi.org/10.1186/s12916-019-1466-7.
Huang Y, Li W, Macheret F, Gabriel RA, Ohno-Machado L. A tutorial on calibration measurements and calibration models for clinical prediction models. J Am Med Informatics Assoc. 2020;27(4):621–33. https://doi.org/10.1093/jamia/ocz228.
Combrisson E, Jerbi K. Exceeding chance level by chance: the caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy. J Neurosci Methods. 2015;250:126–36. https://doi.org/10.1016/j.jneumeth.2015.01.010.
Arbabshirani MR, Plis S, Sui J, Calhoun VD. Single subject prediction of brain disorders in neuroimaging: promises and pitfalls. Neuroimage. 2017;145:137–65. https://doi.org/10.1016/j.neuroimage.2016.02.079.
Collins GS, de Groot JA, Dutton S, et al. External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol. 2014;14(1):40. https://doi.org/10.1186/1471-2288-14-40.
Christodoulou E, Ma J, Collins GS, Steyerberg EW, Verbakel JY, Van Calster B. A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. J Clin Epidemiol. 2019;110:12–22. https://doi.org/10.1016/j.jclinepi.2019.02.004.
Tivadar RI, Murray MM. A primer on electroencephalography and event-related potentials for organizational neuroscience. Organ Res Methods. 2019;22(1):69–94. https://doi.org/10.1177/1094428118804657.
Blankertz B, Tomioka R, Lemm S, Kawanabe M, Muller K. Optimizing spatial filters for robust EEG single-trial analysis. IEEE Signal Process Mag. 2008;25(1):41–56. https://doi.org/10.1109/MSP.2008.4408441.
Rivet B, Souloumiac A, Attina V, Gibert G. xDAWN algorithm to enhance evoked potentials: application to brain-computer interface. IEEE Trans Biomed Eng. 2009;56(8):2035–43. https://doi.org/10.1109/TBME.2009.2012869.
Grosse-Wentrup M, Liefhold C, Gramann K, Buss M. Beamforming in noninvasive brain-computer interfaces. IEEE Trans Biomed Eng. 2009;56(4):1209–19. https://doi.org/10.1109/TBME.2008.2009768.
Magnetoencephalography SS. Basic principles. Ann Indian Acad Neurol. 2014;17(5):107. https://doi.org/10.4103/0972-2327.128676.
Bennington JY, Polich J. Comparison of P300 from passive and active tasks for auditory and visual stimuli. Int J Psychophysiol. 1999;34(2):171–7. https://doi.org/10.1016/S0167-8760(99)00070-7.
Dunning JP, Hajcak G. See no evil: directing visual attention within unpleasant images modulates the electrocortical response. Psychophysiology. 2009;46(1):28–33. https://doi.org/10.1111/j.1469-8986.2008.00723.x.
Hajcak G, MacNamara A, Foti D, Ferri J, Keil A. The dynamic allocation of attention to emotion: simultaneous and independent evidence from the late positive potential and steady state visual evoked potentials. Biol Psychol. 2013;92(3):447–55. https://doi.org/10.1016/j.biopsycho.2011.11.012.
Kam JWY, Xu J, Handy TC. I don’t feel your pain (as much): the desensitizing effect of mind wandering on the perception of others’ discomfort. Cogn Affect Behav Neurosci. 2014;14(1):286–96. https://doi.org/10.3758/s13415-013-0197-z.
Thielen J, Bosch SE, van Leeuwen TM, van Gerven MAJ, van Lier R. Evidence for confounding eye movements under attempted fixation and active viewing in cognitive neuroscience. Sci Rep. 2019;9(1):17456. https://doi.org/10.1038/s41598-019-54018-z.
Ramspek CL, Jager KJ, Dekker FW, Zoccali C, van Diepen M. External validation of prognostic models: what, why, how, when and where? Clin Kidney J. 2021;14(1):49–58. https://doi.org/10.1093/ckj/sfaa188.
Hutson M. Artificial intelligence faces reproducibility crisis. Science (80-). 2018;359(6377):725–6. https://doi.org/10.1126/science.359.6377.725.
Blankertz B, Lemm S, Treder M, Haufe S, Müller K-R. Single-trial analysis and classification of ERP components—a tutorial. Neuroimage. 2011;56(2):814–25. https://doi.org/10.1016/j.neuroimage.2010.06.048.
Mende-Siedlecki P, Lin J, Ferron S, Gibbons C, Drain A, Goharzad A. Seeing no pain: assessing the generalizability of racial bias in pain perception. Emotion. 2021;21(5):932–50. https://doi.org/10.1037/emo0000953.
Osborn J, Derbyshire SWG. Pain sensation evoked by observing injury in others. Pain. 2010;148(2):268–74. https://doi.org/10.1016/j.pain.2009.11.007.
Eroğlu K, Kayıkçıoğlu T, Osman O. Effect of brightness of visual stimuli on EEG signals. Behav Brain Res. 2020;382:112486. https://doi.org/10.1016/j.bbr.2020.112486.
Cao Y, Contreras-Huerta LS, McFadyen J, Cunnington R. Racial bias in neural response to others’ pain is reduced with other-race contact. Cortex. 2015;70:68–78. https://doi.org/10.1016/j.cortex.2015.02.010.
Bas-Sarmiento P, Fernández-Gutiérrez M, Baena-Baños M, Correro-Bermejo A, Soler-Martins PS, de la Torre-Moyano S. Empathy training in health sciences: A systematic review. Nurse Educ Pract. 2020;44:102739. https://doi.org/10.1016/j.nepr.2020.102739.
Fan J, Upadhye S, Worster A. Understanding receiver operating characteristic (ROC) curves. CJEM. 2006;8(01):19–20. https://doi.org/10.1017/S1481803500013336.