The reliability of eyetracking to assess attentional bias to threatening words in healthy individuals
Tóm tắt
Eyetracking is commonly used to investigate attentional bias. Although some studies have investigated the internal consistency of eyetracking, data are scarce on the test–retest reliability and agreement of eyetracking to investigate attentional bias. This study reports the test–retest reliability, measurement error, and internal consistency of 12 commonly used outcome measures thought to reflect the different components of attentional bias: overall attention, early attention, and late attention. Healthy participants completed a preferential-looking eyetracking task that involved the presentation of threatening (sensory words, general threat words, and affective words) and nonthreatening words. We used intraclass correlation coefficients (ICCs) to measure test–retest reliability (ICC > .70 indicates adequate reliability). The ICCs(2, 1) ranged from –.31 to .71. Reliability varied according to the outcome measure and threat word category. Sensory words had a lower mean ICC (.08) than either affective words (.32) or general threat words (.29). A longer exposure time was associated with higher test–retest reliability. All of the outcome measures, except second-run dwell time, demonstrated low measurement error (<6%). Most of the outcome measures reported high internal consistency (α > .93). Recommendations are discussed for improving the reliability of eyetracking tasks in future research.
Tài liệu tham khảo
Amir, N., Beard, C., Burns, M., & Bomyea, J. (2009). Attention modification program in individuals with generalized anxiety disorder. Journal of Abnormal Psychology, 118, 28–33. doi:10.1037/a0012589
Amir, N., Weber, G., Beard, C., Bomyea, J., & Taylor, C. T. (2008). The effect of a single-session attention modification program on response to a public-speaking challenge in socially anxious individuals. Journal of Abnormal Psychology, 117, 860–868. doi:10.1037/a0013445
Armstrong, T., & Olatunji, B. O. (2012). Eye tracking of attention in the affective disorders: A meta-analytic review and synthesis. Clinical Psychology Review, 32, 704–723. doi:10.1016/j.cpr.2012.09.004
Bar-Haim, Y., Lamy, D., Pergamin, L., Bakermans-Kranenburg, M. J., & van IJzendoorn, M. H. (2007). Threat-related attentional bias in anxious and nonanxious individuals: A meta-analytic study. Psychological Bulletin, 133, 1–24. doi:10.1037/0033-2909.133.1.1
Bartko, J. J. (1991). Measurement and reliability: Statistical thinking considerations. Schizophrenia Bulletin, 17, 483–489. doi:10.1093/schbul/17.3.483
Charter, R. A., & Feldt, L. S. (2001). Meaning of reliability in terms of correct and incorrect clinical decisions: The art of decision making is still alive. Journal of Clinical and Experimental Neuropsychology, 23, 530–537. doi:10.1076/jcen.23.4.530.1227
Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284–290. doi:10.1037/1040-3590.6.4.284
Crombez, G., Van Ryckeghem, D. M., Eccleston, C., & Van Damme, S. (2013). Attentional bias to pain-related information: A meta-analysis. Pain, 154, 497–510. doi:10.1016/j.pain.2012.11.013
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334. doi:10.1007/BF02310555
de Vet, H. C., Terwee, C. B., Knol, D. L., & Bouter, L. M. (2006). When to use agreement versus reliability measures. Journal of Clinical Epidemiology, 59, 1033–1039. doi:10.1016/j.jclinepi.2005.10.015
de Vet, H. C., Terwee, C. B., Mokkink, L. B., & Knol, D. L. (2011). Measurement in medicine: A practical guide (1st ed.). Cambridge: Cambridge University Press.
Dear, B. F., Sharpe, L., Nicholas, M. K., & Refshauge, K. (2011). The psychometric properties of the dot-probe paradigm when used in pain-related attentional bias research. Journal of Pain, 12, 1247–1254. doi:10.1016/j.jpain.2011.07.003
Dehghani, M., Sharpe, L., & Nicholas, M. K. (2003). Selective attention to pain-related information in chronic musculoskeletal pain patients. Pain, 105, 37–46. doi:10.1016/s0304-3959(03)00224-0
Donaldson, C., Lam, D., & Mathews, A. (2007). Rumination and attention in major depression. Behaviour Research and Therapy, 45, 2664–2678. doi:10.1016/j.brat.2007.07.002
Duque, A. (2015). Double attention bias for positive and negative emotional faces in clinical depression: Evidence from an eye-tracking study. Journal of Behavior Therapy and Experimental Psychiatry, 46, 107–114. doi:10.1016/j.jbtep.2014.09.005
Farzin, F., Scaggs, F., Hervey, C., Berry-Kravis, E., & Hessl, D. (2011). Reliability of eye tracking and pupillometry measures in individuals with fragile X syndrome. Journal of Autism and Developmental Disorders, 41, 1515–1522. doi:10.1007/s10803-011-1176-2
Fashler, S. R., & Katz, J. (2014). More than meets the eye: Visual attention biases in individuals reporting chronic pain. Journal of Pain Research, 7, 557–570. doi:10.2147/JPR.S67431
Felmingham, K. L., Rennie, C., Manor, B., & Bryant, R. A. (2011). Eye tracking and physiological reactivity to threatening stimuli in posttraumatic stress disorder. Journal of Anxiety Disorders, 25, 668–673. doi:10.1016/j.janxdis.2011.02.010
Gao, X., Wang, Q., Jackson, T., Zhao, G., Liang, Y., & Chen, H. (2011). Biases in orienting and maintenance of attention among weight dissatisfied women: An eye-movement study. Behaviour Research and Therapy, 49, 252–259. doi:10.1016/j.brat.2011.01.009
Giraudeau, B., & Mary, J. Y. (2001). Planning a reproducibility study: How many subjects and how many replicates per subject for an expected width of the 95 per cent confidence interval of the intraclass correlation coefficient. Statistical Medicine, 20, 3205–3214. doi:10.1002/sim.935
Gliner, J. A., Morgan, G. A., & Harmon, R. J. (2001). Measurement reliability. Journal of the American Academy of Child and Adolescent Psychiatry, 40, 486–488. doi:10.1097/00004583-200104000-00019
Guasch, M., Boada, R., Ferré, P., & Sánchez-Casas, R. (2013). NIM: A Web-based Swiss army knife to select stimuli for psycholinguistic studies. Behavior Research Methods, 45, 765–771. doi:10.3758/s13428-012-0296-8
Guyatt, G., Walter, S., & Norman, G. (1987). Measuring change over time: Assessing the usefulness of evaluative instruments. Journal of Chronic Diseases, 40, 171–178. doi:10.1016/0021-9681(87)90069-5
Jensen, M. P., Johnson, L. E., Gertz, K. J., Galer, B. S., & Gammaitoni, A. R. (2013). The words patients use to describe chronic pain: Implications for measuring pain quality. Pain, 154, 2722–2728. doi:10.1016/j.pain.2013.08.003
Kimble, M. O., Fleming, K., Bandy, C., Kim, J., & Zambetti, A. (2010). Eye tracking and visual attention to threating stimuli in veterans of the Iraq war. Journal of Anxiety Disorders, 24, 293–299. doi:10.1016/j.janxdis.2009.12.006
Kopriva, R. J., & Shaw, D. G. (1991). Power estimates: The effect of dependent variable reliability on the power of one-factor ANOVAs. Educational and Psychological Measurement, 51, 585–595. doi:10.1177/0013164491513006
Kottner, J., & Streiner, D. L. (2011). The difference between reliability and agreement. Journal of Clinical Epidemiology, 64, 701–702. doi:10.1016/j.jclinepi.2010.12.001
Lakes, K. D. (2013). Restricted sample variance reduces generalizability. Psychological Assessment, 25, 643–650. doi:10.1037/a0030912
Lazarov, A., Abend, R., & Bar-Haim, Y. (2016). Social anxiety is related to increased dwell time on socially threatening faces. Journal of Affective Disorders, 193, 282–288. doi:10.1016/j.jad.2016.01.007
Liossi, C., Schoth, D. E., Godwin, H. J., & Liversedge, S. P. (2014). Using eye movements to investigate selective attention in chronic daily headache. Pain, 155, 503–510. doi:10.1016/j.pain.2013.11.014
Liversedge, & Findlay, J. M. (2000). Saccadic eye movements and cognition. Trends in Cognitive Sciences, 4, 6–14. doi:10.1016/S1364-6613(99)01418-7
Loken, E., & Gelman, A. (2017). Measurement error and the replication crisis. Science, 355, 584–585. doi:10.1126/science.aal3618
Lovibond, S. H., & Lovibond, P. F. (1995). Manual for the Depression Anxiety Stress Scales (2nd ed.). Sydney: Psychology Foundation.
Marian, V., Blumenfeld, H. K., & Kaushanskaya, M. (2007). The Language Experience and Proficiency Questionnaire (LEAP-Q): Assessing language profiles in bilinguals and multilinguals. Journal of Speech, Language, and Hearing Research, 50, 940–967. doi:10.1044/1092-4388(2007/067)
Meyer, J. P. (2010). Reliability. New York: Oxford University Press.
Mogg, K., Bradley, B. P., Field, M., & De Houwer, J. (2003). Eye movements to smoking-related pictures in smokers: Relationship between attentional biases and implicit and explicit measures of stimulus valence. Addiction, 98, 825–836. doi:10.1046/j.1360-0443.2003.00392.x
Mogg, K., Bradley, B., Miles, F., & Dixon, R. (2004). Time course of attentional bias for threat scenes: Testing the vigilance–avoidance hypothesis. Cognition and Emotion, 18, 689–700. doi:10.1080/02699930341000158
Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., … & de Vet, H. C. W. (2010). The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. Journal of Clinical Epidemiology, 63, 737–745. doi:10.1016/j.jclinepi.2010.02.006
Nunnally, J. C. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.
Price, R. B., Kuckertz, J. M., Siegle, G. J., Ladouceur, C. D., Silk, J. S., Ryan, N. D., … & Amir, N. (2015). Empirical recommendations for improving the stability of the dot-probe task in clinical research. Psychological Assessment, 27, 365–376. doi:10.1037/pas0000036
Radach, R., & Kennedy, A. (2004). Theoretical perspectives on eye movements in reading: Past controversies, current issues, and an agenda for future research. European Journal of Cognitive Psychology, 16, 3–26. doi:10.1080/09541440340000295
Rayner, K. (1989). In A. Pollatsek (Ed.), Psychology of reading. New Jersey: Prentice Hall.
Rodebaugh, T. L., Scullin, R. B., Langer, J. K., Dixon, D. J., Huppert, J. D., Bernstein, A., … & Lenze, E. J. (2016). Unreliability as a threat to understanding psychopathology: The cautionary tale of attentional bias. Journal of Abnormal Psychology, 125, 840–851. doi:10.1037/abn0000184
Schmukle, S. C. (2005). Unreliability of the dot probe task. European Journal of Personality, 19, 595–605. doi:10.1002/per.554
Scholtes, V. A., Terwee, C. B., & Poolman, R. W. (2011). What makes a measurement instrument valid and reliable? Injury, 42, 236–240. doi:10.1016/j.injury.2010.11.042
Sharpe, L., Haggman, S., Nicholas, M., Dear, B. F., & Refshauge, K. (2014). Avoidance of affective pain stimuli predicts chronicity in patients with acute low back pain. Pain, 155, 45–52. doi:10.1016/j.pain.2013.09.004
Sharpe, L., Ianiello, M., Dear, B. F., Nicholson Perry, K., Refshauge, K., & Nicholas, M. K. (2012). Is there a potential role for attention bias modification in pain patients? Results of 2 randomised, controlled trials. Pain, 153, 722–731. doi:10.1016/j.pain.2011.12.014
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428. doi:10.1037/0033-2909.86.2.420
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. doi:10.1177/0956797611417632
Open Science Framework. (MT3K8). Skinner, I. W., Hübscher, M., Moseley, G. L., Lee, H., Traeger, A., Wand, B. M., Gustin, S. M., & McAuley, J. (Eds.) (2016) The test- retest reliability of eye tracking to measure attentional bias. Retrieved from http://www.osf.io/mt3k8 doi:10.17605/OSF.IO/MT3K8
Streiner, D. L. (2003a). Being inconsistent about consistency: When coefficient alpha does and doesn’t matter. Journal of Personality Assessment, 80, 217–222. doi:10.1207/S15327752JPA8003_01
Streiner, D. L. (2003b). Health measurement scales: a practical guide to their development and use (3rd ed.). Oxford: Oxford University Press.
Streiner, D. L. (2003c). Starting at the beginning: An introduction to coefficient alpha and internal consistency. Journal of Personality Assessment, 80, 99–103. doi:10.1207/S15327752JPA8001_18
Sullivan, M., Bishop, S., & Pivik, J. (1995). The Pain Catastrophizing Scale: Development and validation. Psychological Assessment, 7, 524–532. doi:10.1037/1040-3590.7.4.524
Todd, J., Sharpe, L., Johnson, A., Nicholson Perry, K., Colagiuri, B., & Dear, B. F. (2015). Towards a new model of attentional biases in the development, maintenance, and management of pain. Pain, 156, 1589–1600. doi:10.1097/j.pain.0000000000000214
Todd, J., Sharpe, L., & Colagiuri, B. (2016). Attentional bias modification and pain: The role of sensory and affective stimuli. Behaviour Research and Therapy, 83, 53–61. doi:10.1016/j.brat.2016.06.002
Toh, W. L., Rossell, S. L., & Castle, D. J. (2011). Current visual scanpath research: A review of investigations into the psychotic, anxiety, and mood disorders. Comprehensive Psychiatry, 52, 567–579. doi:10.1016/j.comppsych.2010.12.005
Vazquez, C., Blanco, I., Sanchez, A., & McNally, R. J. (2016). Attentional bias modification in depression through gaze contingencies and regulatory control using a new eye-tracking intervention paradigm: Study protocol for a placebo-controlled trial. BMC Psychiatry, 16, 439. doi:10.1186/s12888-016-1150-9
Vervoort, T., Trost, Z., Prkachin, K. M., & Mueller, S. C. (2013). Attentional processing of other’s facial display of pain: An eye tracking study. Pain, 154, 836–844. doi:10.1016/j.pain.2013.02.017
Waechter, S., Nelson, A. L., Wright, C., Hyatt, A., & Oakman, J. (2014). Measuring attentional bias to threat: Reliability of dot probe and eye movement indices. Cognitive Therapy and Research, 38, 313–333. doi:10.1007/s10608-013-9588-2
Weir, J. P. (2005). Quantifying test–retest reliability using the intraclass correlation coefficient and the SEM. Journal of Strength and Conditioning Research, 19, 231–240. doi:10.1519/15184.1
White, L. K., Suway, J. G., Pine, D. S., Bar-Haim, Y., & Fox, N. A. (2011). Cascading effects: The influence of attention bias to threat on the interpretation of ambiguous information. Behaviour Research and Therapy, 49, 244–251. doi:10.1016/j.brat.2011.01.004
Yang, Z., Jackson, T., Gao, X., & Chen, H. (2012). Identifying selective visual attention biases related to fear of pain by tracking eye movements within a dot-probe paradigm. Pain, 153, 1742–1748. doi:10.1016/j.pain.2012.05.011
Yang, Z., Jackson, T., & Chen, H. (2013). Effects of chronic pain and pain-related fear on orienting and maintenance of attention: an eye movement study. Journal of Pain, 14, 1148–1157. doi:10.1016/j.jpain.2013.04.017