Effect of Item Order on Certain Psychometric Properties: A Demonstration on a Cyberloafing Scale

Frontiers in Psychology - Tập 12
Murat Doğan Şahin1
1Department of Educational Sciences, Faculty of Education, Anadolu University, Eskişehir, Turkey

Tóm tắt

Many studies have been conducted on the effect of item order in self-report questionnaires on mean scores. This research aims to study the effect of item order on measurement invariance in addition to mean scores. To this end, two groups randomly obtained from the same sample were presented a fixed order form in which all items belonging to the same dimension were adjacent to each other, and a random order form in which the items were randomly sequenced respectively. The results obtained revealed a statistically significant difference between the mean scores of the two forms. In the next stage of the study, the fit indices obtained from the confirmatory factor analysis (CFA) applied to the two separate forms and the modification indices (MI) suggested by the software were compared. Both forms returned high modification suggestions for adjacent items or items presented near each other. Additionally, it was found that high χ2 reductions suggested by the MIs in one form resulted in low χ2 reductions in the other. Lastly, multiple group CFA (mg-CFA) was conducted to determine whether or not measurement invariance was achieved through different item order presentations of the scale. The findings indicate that measurement invariance could not be achieved even at the first stage of analysis. It may specifically be stated that presenting respondents items under the same dimension together ensures empirical findings congruent with theoretical structure.

Từ khóa


Tài liệu tham khảo

Akbulut, 2015, Predictors of inconsistent responding in web surveys, Internet Res., 25, 131, 10.1108/IntR-01-2014-0017

Akbulut, 2017, Cyberloafing and social desirability bias among students and employees, Comput. Hum. Behav., 72, 87, 10.1016/j.chb.2017.02.043

Akbulut, 2016, In search of a measure to investigate cyberloafing in educational settings, Comput. Hum. Behav., 55, 616, 10.1016/j.chb.2015.11.002

Askew, 2014, Explaining cyberloafing: the role of the theory of planned behavior, Comput. Hum. Behav., 36, 510, 10.1016/j.chb.2014.04.006

Bentler, 1980, Significance tests and goodness of fit in the analysis of covariance structures, Psychol. Bull., 88, 588, 10.1037/0033-2909.88.3.588

Bowman, 2014, Effects of item order and response options in college student surveys, Methodological advances and issues in studying college impact: New directions for institutional research, 99

Chen, 2010

Dillman, 2000, Mail and internet surveys: The tailored design method

Dimitrov, 2010, Testing for factorial invariance in the context of construct validation, Meas. Eval. Couns. Dev., 43, 121, 10.1177/0748175610373459

Dursun, 2018, Predictors of cyberloafing among preservice information technology teachers, Contemp. Educ. Technol., 9, 22, 10.30935/cedtech/6209

Fraenkel, 2009, How to design and evaluate research in education

Gökçearslan, 2018, Smartphone addiction, cyberloafing, stress and social support among university students: a path analysis, Child. Youth Serv. Rev., 91, 47, 10.1016/j.childyouth.2018.05.036

Hamby, 2016, Survey satisficing inflates reliability and validity measures: an experimental comparison of college and amazon mechanical turk samples, Educ. Psychol. Meas., 76, 912, 10.1177/0013164415627349

Harrison, 1993, Cognitive processes inself-report responses: tests of item context effects in work attitude measures, J. Appl. Psychol., 78, 129, 10.1037/0021-9010.78.1.129

Hooper, 2008, Structural equation modelling: guidelines for determining model fit, Electron. J. Bus. Res. Methods, 6, 53

Hu, 1999, Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives, Struct. Equ. Model., 6, 1, 10.1080/10705519909540118

Huang, 2015, The impact of definition and question order on the prevalence of bullying victimization using student self-reports, Psychol. Assess., 27, 1484, 10.1037/pas0000149

Huang, 2016, Question order affects the measurement of bullying victimization among middle school students, Educ. Psychol. Meas., 76, 724, 10.1177/0013164415622664

Huang, 2012, Detecting and deterring insufficient effort responding to surveys, J. Bus. Psychol., 27, 99, 10.1007/s10869-011-9231-8

Kaplan, 2013, General and specific question sequence effects in satisfaction surveys: integrating directional and correlational effects, J. Happiness Stud., 14, 1443, 10.1007/s10902-012-9388-5

Kian-Yeik, 2018, Assessing cyberloafing behaviour among university students: a validation of the cyberloafing scale, Pertanika J. Soc. Sci. Hum., 26, 409, 10.5210/fm.v23i6.7837

King, 2000, Social desirability bias: a neglected aspect of validity testing, Psychol. Mark., 17, 79, 10.1002/(SICI)1520-6793(200002)17:2<79::AID-MAR2>3.0.CO;2-0

Kline, 2011, Principles and practice of structural equation modeling

Knowles, 1988, Item context effects on personality scales: measuring changes the measure, J. Pers. Soc. Psychol., 55, 312, 10.1037/0022-3514.55.2.312

Korkmaz, 2014, MVN: an R package for assessing multivariate normality, R J., 6, 151, 10.32614/RJ-2014-031

Krosnick, 1996, Satisficing in surveys: initial evidence, New Direct. Program Eval., 70, 29, 10.1002/ev.1033

Larson, 2019, Controlling social desirability bias, Int. J. Mark. Res., 61, 534, 10.1177/1470785318805305

Lasorsa, 2003, Question-order effects in surveys: the case of political interest, news attention, and knowledge, J. Mass Commun. Q., 80, 499, 10.1177/107769900308000302

Marsh, 2015, Invariance testing across samples and time: cohort sequence analysis of perceived body composition, An introduction to intermediate and advanced statistical analyses for sport and exercise scientists, 121

McFarland, 1981, Effects of question order on survey responses, Public Opin. Q., 45, 208, 10.1086/268651

Meade, 2012, Identifying careless responses in survey data, Psychol. Methods, 17, 437, 10.1037/a0028085

Moody, 2013, Using the theory of interpersonal behavior to explain non work related personal use of the internet at work, Inf. Manag., 50, 322, 10.1016/j.im.2013.04.005

Muthén, , Mplus user’s guide

Nederhof, 1985, Methods of coping with social desirability bias: a review, Eur. J. Soc. Psychol., 15, 263, 10.1002/ejsp.2420150303

Peltier, 1990, An investigation of response bias in the chapman scale, Educ. Psychol. Meas., 50, 803, 10.1177/0013164490504008

Phillips, 1972, Some effects of "social desirability" in survey studies, Am. J. Soc., 77, 921, 10.1086/225231

Putnick, 2016, Measurement invariance conventions and reporting: the state of the art and future directions for psychological research, Dev. Rev., 41, 71, 10.1016/j.dr.2016.06.004

Saeki, 2013, Life satisfaction judgments and item-order effects across cultures, Soc. Indic. Res., 118, 941, 10.1007/s11205-013-0463-z

Schell, 2013, Item grouping and item randomization in personality measurement, Pers. Individ. Differ., 55, 317, 10.1016/j.paid.2013.03.008

Schumacker, 1996, A beginner’s guide to structural equation modeling

Schuman, 1997, Questions about attitude survey questions, Sociological methodology, 1973

Schuman, 1996, Questions and answers in attitude surveys: Experiments on question form, wording, and context

Schwarz, 1999, Self-reports: how the questions shape the answers, Am. Psychol., 54, 93, 10.1037/0003-066X.54.2.93

Schwarz, 2007, Cognitive aspects of survey methodology, Appl. Cogn. Psychol., 21, 277, 10.1002/acp.1340

Şen, 2020, Mplus ile yapısal eşitlik modellemesi uygulamaları

Shorey, 2016, The effect of item order on psychological aggression reporting: an examination with the multidimensional measure of emotional abuse, Partn. Abus., 7, 125, 10.1891/1946-6560.7.2.125

Sivrikova, 2019, Assessing of use of the internet for personal reasons at lessons at school: a validation of the cyberloafing scale, SHS Web Conf., 70, 06010, 10.1051/shsconf/20197006010

Strack, 1988, Priming and communication: social determinants of information use in judgments of life satisfaction, Eur. J. Soc. Psychol., 18, 429, 10.1002/ejsp.2420180505

2020

Tourangeau, 1988, Cognitive processes underlying context effects in attitude measurement, Psychol. Bull., 103, 299, 10.1037/0033-2909.103.3.299

Van De Schoot, 2015, Editorial: measurement invariance, Front. Psychol., 6, 1064, 10.3389/fpsyg.2015.01064

Vandenberg, 2000, A review and synthesis of the measurement invariance literature: suggestions, practices, and recommendations for organizational research, Organ. Res. Methods, 3, 4, 10.1177/109442810031002

Wagner, 2012, Lostsleepand cyberloafing: evidence from the laboratory and a day light saving time quasi-experiment, J. Appl. Psychol., 97, 1068, 10.1037/a0027557

Weatherbee, 2010, Counterproductive use of technology at work: information & communications technologies and cyberdeviancy, Hum. Resour. Manag. Rev., 20, 35, 10.1016/j.hrmr.2009.03.012

Weinberg, 2018, The measurement of bubjective wellbeing: item-order effects in the personal wellbeing index—adult, J. Happiness Stud., 19, 315, 10.1007/s10902-016-9822-1

Widaman, 1997, Exploring the measurement invariance of psychological instruments: applications in the substance use domain, The science of prevention: Methodological advances from alcohol and substance abuse research, 281, 10.1037/10222-009

Wu, 2018, Student cyberloafing in and out of the classroom in China and the relationship with student performance, Cyberpsychol. Behav. Soc. Netw., 21, 199, 10.1089/cyber.2017.0397

Zhang, 2013, Speeding in web surveys: the tendency to answer very fast and its association with straightlining, Survey Res. Methods, 8, 127, 10.18148/srm/2014.v8i2.5453

Zhao, 2008, Adult metacomprehension: judgment processes and accuracy constraints, Educ. Psychol. Rev., 20, 191, 10.1007/s10648-008-9073-8

Thông tin
Thông tin xuất bản

Frontiers in Psychology

Tập 12

Thông tin tác giả