The failing measurement of attitudes: How semantic determinants of individual survey responses come to replace measures of attitude strength

Abdi, H. (2003). Factor rotations in factor analysis. In M. Lewis-Beck, A. Bryman, & T. Futing (Eds.), Encyclopedia of social sciences research methods (pp. 792–795). Thousand Oaks, CA: Sage. Aiken, L. S., West, S. G., & Reno, R. R. (1991). Multiple regression: Testing and interpreting interactions. Thousand Oaks, CA: Sage. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association. Andrich, D. (1996). A hyperbolic cosine latent trait model for unfolding polytomous responses: Reconciling Thurstone and Likert methodologies. British Journal of Mathematical and Statistical Psychology, 49, 347–365. Antonakis, J., Bendahan, S., Jacquart, P., & Lalive, R. (2010). On making causal claims: A review and recommendations. The Leadership Quarterly, 21, 1086–1120. Arnulf, J. K., & Larsen, K. R. (2015). Overlapping semantics of leadership and heroism: Expectations of omnipotence, identification with ideal leaders and disappointment in real managers. Scandinavian Psychologist, 2, 1–28. doi:https://doi.org/10.15714/scandpsychol.2.e3 Arnulf, J. K., Larsen, K. R., Martinsen, Ø. L., & Bong, C. H. (2014). Predicting survey responses: How and why semantics shape survey statistics in organizational behavior. PLoS ONE, 9, e106361:1–13. doi:https://doi.org/10.1371/journal.pone.0106361 Arnulf, J. K., Larsen, K. R., Martinsen, Ø. L., & Bong, C. H. (2015). A new approach to psychological measures in leadership research. Scandinavian Psychologist, 2, e4. doi:https://doi.org/10.15714/scandpsychol.2.e4 Avolio, B. J., Bass, B. M., & Jung, D. I. (1995). Multifactor Leadership Questionnaire (Technical Report). Redwood City, CA: Mind Garden. Bagozzi, R. P. (2011). Measurement and meaning in information systems and organizational research: Methodological and philosophical foundations. MIS Quarterly, 35, 261–292. Bernerth, J. B., Armenakis, A. A., Feild, H. S., Giles, W. F., & Walker, H. J. (2007). Leader–member social exchange (LMSX): Development and validation of a scale. Journal of Organizational Behavior, 28, 979–1003. doi:https://doi.org/10.1002/job.443 Borsboom, D. (2008). Latent variable theory. Measurement, 6, 25–53. Borsboom, D. (2009). Educational measurement [Book review]. Structural Equation Modeling, 16, 702–711. doi:https://doi.org/10.1080/10705510903206097 Brennan, R. L. (2010). Generalizability theory. In P. P. B. McGaw (Ed.), International encyclopedia of education (3rd ed., pp. 61–68). Oxford, UK: Elsevier. Cammann, C., Fichman, M., Jenkins, D., & Klesh, J. (1979). The Michigan organizational assessment questionnaire. Ann Arbor, MI: University of Michigan. Cascio, W. F. (2012). Methodological issues in international HR management research. International Journal of Human Resource Management, 23, 2532–2545. doi:https://doi.org/10.1080/09585192.2011.561242 Conger, J. A., & Kanungo, R. N. (1998). Charismatic leadership in organizations. Thousand Oaks, CA: Sage. Coombs, C. H., & Kao, R. C. (1960). On a connection between factor analysis and multidimensional unfolding. Psychometrika, 25, 219–231. Costa, P. T., & McCrae, R. R. (1992). Revised NEO personality Inventory: Professional manual. Odessa, FL: Psychological Assessment Resources. Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302. Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13, 319–340. doi:https://doi.org/10.2307/249008 de Vries, J., Byrne, M., & Kehoe, E. (2015). Cognitive dissonance induction in everyday life: An fMRI study. Social Neuroscience, 10, 268–281. doi:https://doi.org/10.1080/17470919.2014.990990 Deerwester, S., Dumais, S., Furnas, G., Landauer, T., & Harshman, R. (1990). Indexing by Latent Semantic Analysis. Journal of the American Society for Information Science, 41, 391–407. Dennis, S., Landauer, T., Kintsch, W., & Quesada, J. (2003). Introduction to latent semantic analysis. Paper presented at the 25th Annual Meeting of the Cognitive Science Society, Boston, MA. Dennis, S., Landauer, T., Kintsch, W., & Quesada, J. (2013). Introduction to latent semantic analysis. Boulder, CO: University of Colorado. Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272–299. Feldman, J. M., & Lynch, J. G. J. (1988). Self-generated validity and other effects of measurement on belief, attitude, intention, and behavior. Journal of Applied Psychology, 73, 421–435. Felfe, J., & Schyns, B. (2014). Romance of leadership and motivation to lead. Journal of Managerial Psychology, 29, 850–865. doi:https://doi.org/10.1108/jmp-03-2012-0076 Fishbein, M., & Ajzen, I. (1975). Belief, attitude, intention and behavior: An introduction to theory and research. Reading, MA: Addison-Wesley. Fisher, H. E., Island, H. D., Rich, J., Marchalik, D., & Brown, L. L. (2015). Four broad temperament dimensions: Description, convergent validation correlations, and comparison with the Big Five. Frontiers in Psychology, 6, 1098. doi:https://doi.org/10.3389/fpsyg.2015.01098 Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18, 39–50. Gefen, D., & Larsen, K. R. (2017). Controlling for lexical closeness in survey research: A demonstration on the Technology Acceptance Model. Journal of the Association for Information Systems, 18(10), 1. Gollwitzer, P., & Sheeran, P. (2006). Implementation intentions and goal achievement: A meta-analysis of effects and processes. Advances in Experimental Social Psychology, 38, 69–118. Honey, C. J., Thomson, C. R., Lerner, Y., & Hasson, U. (2012). Not lost in translation: Neural responses shared across languages. Journal of Neuroscience, 32, 15277–15283. Ilies, R., Nahrgang, J. D., & Morgeson, F. P. (2007). Leader–member exchange and citizenship behaviors: A meta-analysis. Journal of Applied Psychology, 92, 269–277. Jackson, R. L., Hoffman, P., Pobric, G., & Ralph, M. A. L. (2016). The semantic network at work and rest: Differential connectivity of anterior temporal lobe subregions. Journal of Neuroscience, 36, 1490–1501. doi:https://doi.org/10.1523/jneurosci.2999-15.2016 Jöreskog, K. G. (1993). Testing structural equation models. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 294–316). Newbury Park, CA: Sage. Jöreskog, K. G., & Sörbom, D. (1993). LISREL 8: Structural equation modeling with the SIMPLIS command language. Chicago, IL: SSI Scientific Software. Kahneman, D. (2011). Thinking, fast and slow. New York, NY: Farrar, Straus & Giroux. Kern, M. L., Eichstaedt, J. C., Schwartz, H. A., Park, G., Ungar, L. H., Stillwell, D. J., … Seligman, M. E. P. (2014). From “sooo excited!!!” to “so proud”: Using language to study development. Developmental Psychology, 50, 178–188. doi:https://doi.org/10.1037/a0035048 Kim, S., Hahn, H. J., & Lee, J. (2015). Organizational attitudes as precursors to training performance. Human Resource Development Quarterly, 26, 409–429. doi:https://doi.org/10.1002/hrdq.21218 Kline, P. (2000). Handbook of psychological testing (2nd ed.). London, UK: Routledge. Kosinski, M., Bachrach, Y., Kasneci, G., Van-Gael, J., & Graepel, T. (2012). Crowd IQ: measuring the intelligence of crowdsourcing platforms. Proceedings of the 4th Annual ACM Web Science Conference, Evanston, Illinois, June 22–24 (pp. 151–160). New York, NY: ACM. doi:https://doi.org/10.1145/2380718.2380739 Kosinski, M., Stillwell, D., & Graepel, T. (2013). Private traits and attributes are predictable from digital records of human behavior. Proceedings of the National Academy of Sciences, 110, 5802–5805. doi:https://doi.org/10.1073/pnas.1218772110 Krueger, J. (2001). Null hypothesis significance testing: On the survival of a flawed method. American Psychologist, 56, 16–26. Kuvaas, B. (2006a). Performance appraisal satisfaction and employee outcomes: Mediating and moderating roles of motivation. International Journal of Human Resource Management, 17, 504–522. Kuvaas, B. (2006b). Work performance, affective commitment, and work motivation: The roles of pay administration and pay level. Journal of Organizational Behavior, 27, 365–385. Kuvaas, B., Arnulf, J. K., & Dysvik, A. (2008). Social and economic exchange perceptions and intrinsic motivation among knowledge workers. Paper presented at the Academy of Management Annual Meeting, Anaheim, CA. Lamiell, J. T. (2013). Statisticism in personality psychologists’ use of trait constructs: What is it? How was it contracted? Is there a cure? New Ideas in Psychology, 31, 65–71. doi:https://doi.org/10.1016/j.newideapsych.2011.02.009 Landauer, T. K. (2007). LSA as a theory of meaning. In T. K. Landauer, D. S. McNamara, S. Dennis, & W. Kintsh (Eds.), Handbook of latent semantic analysis (pp. 3–34). Mahwah, NJ: Erlbaum. LaPiere, R. T. (1934). Attitudes vs. actions. Social Forces, 13, 230–237. LaPointe, L. L. (2013). Paul Broca and the origins of language in the brain. San Diego, CA, US: Plural Publishing. Larsen, K. R., & Bong, C. H. (2013). Construct identity detection. Leeds School Working Paper Series (pp. 1–70). Boulder, CO: University of Colorado, Boulder. Larsen, K. R., & Bong, C. H. (2016). A tool for addressing construct identity in literature reviews and meta-analyses. MIS Quarterly, 40, 529–551. Larsen, K. R., & Monarchi, D. E. (2004). A mathematical approach to categorization and labeling of qualitative data: The latent categorization method. Sociological Methodology, 34, 349–392. Larsen, K. R., Nevo, D., & Rich, E. (2008). Exploring the semantic validity of questionnaire scales. Paper presented at the Proceedings of the 41st Hawaii International Conference on System Sciences, Waikoloa, Hawaii. Li, Y., McLean, D., Bandar, Z. A., O’Shea, J. D., & Crockett, K. (2006). Sentence similarity based on semantic nets and corpus statistics. IEEE Transactions on Knowledge and Data Engineering, 18, 1138–1150. Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology, 140, 1–55. Lynam, D. R., Hoyle, R. H., & Newman, J. P. (2006). The perils of partialling. Assessment, 13, 328–341. doi:https://doi.org/10.1177/1073191106290562 MacKenzie, S. B., Podsakoff, P. M., & Podsakoff, N. P. (2011). Construct measurement and validation procedures in MIS and behavioral research: Integrating new and existing techniques. MIS Quarterly, 35, 293–334. Markowetz, A., Blaszkiewicz, K., Montag, C., Switala, C., & Schlaepfer, T. E. (2014). Psycho-Informatics: Big data shaping modern psychometrics. Medical Hypotheses, 82, 405–411. doi:https://doi.org/10.1016/j.mehy.2013.11.030 Marsh, H. W., Ludtke, O., Muthén, B., Asparouhov, T., Morin, A. J. S., Trautwein, U., & Nagengast, B. (2010). A new look at the big five factor structure through exploratory structural equation modeling. Psychological Assessment, 22, 471–491. doi:https://doi.org/10.1037/a0019227 Martinsen, Ø. L., Nordvik, H., & Østbø, L. (2003). Norske versjoner av NEO PI- R og NEO FFI (Norwegian versions of the NEO PI-R and NEO FFI). Oslo, Norway: Gyldendal Akademisk forlag. Maul, A. (2017). Rethinking traditional methods of survey validation. Measurement: Interdisciplinary Research and Perspectives, 15, 51–69. doi:https://doi.org/10.1080/15366367.2017.1348108 McClelland, D. C., Koestner, R., & Weinberger, J. (1989). How do self-attributed and implicit motives differ. Psychological Review, 96, 690–702. McKenzie, J. (1998). Fundamental flaws in the five factor model: A re-analysis of the seminal correlation matrix from which the “openness-to-experience” factor was extracted. Personality and Individual Differences, 24, 475–480. doi:https://doi.org/10.1016/s0191-8869(97)00223-7 Meyer, J. P., Allen, N. J., & Smith, C. A. (1993). Commitment to organizations and occupations: Extension and test of a three-component conceptualization. Journal of Applied Psychology, 78, 538–551. Michell, J. (1994). Measuring dimensions of belief by unidimensional unfolding. Journal of Mathematical Psychology, 38, 244–273. Michell, J. (2013). Constructs, inferences, and mental measurement. New Ideas in Psychology, 31, 13–21. doi:https://doi.org/10.1016/j.newideapsych.2011.02.004 Mihalcea, R., Corley, C., & Strapparava, C. (2006). Corpus-based and knowledge-based measures of text semantic similarity. Paper presented at the 21st National Conference on Artificial Intelligence, Boston, MA. Miller, G. (1995). WordNet: A lexical database for English. Communications of the ACM, 38, 39–41. doi:https://doi.org/10.1145/219717.219748 Mohler, M., & Mihalcea, R. (2009). Text-to-text semantic similarity for automatic short answer grading. Paper presented at the 12th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2009), Athens, Greece. Nakagawa, J., Takahashi, M., Okada, R., Matsushima, E., & Matsuda, T. (2015). Women’s preference for a male acquaintance enhances social reward processing of material goods in the anterior cingulate cortex. PLoS ONE, 10, e136168:1–14. doi:https://doi.org/10.1371/journal.pone.0136168 Nimon, K., Shuck, B., & Zigarmi, D. (2015). Construct overlap between employee engagement and job satisfaction: A function of semantic equivalence? Journal of Happiness Studies, 17, 1149–1171. doi:https://doi.org/10.1007/s10902-015-9636-6 Nunnally, J. C., & Bernstein, I. H. (2010). Psychometric theory (3rd ed.). New York, NY: McGraw-Hill. Podsakoff, P. M., MacKenzie, S. B., & Podsakoff, N. P. (2012). Sources of method bias in social science research and recommendations on how to control it. Annual Review of Psychology, 63, 539–569. doi:https://doi.org/10.1146/annurev-psych-120710-100452 Podsakoff, P. M., & Organ, D. W. (1986). Self-reports in organizational research—Problems and prospects. Journal of Management, 12, 531–544. Poli, R., Healy, M., & Kameas, A. (2010). WordNet. In C. Fellbaum (Ed.), Theory and applications of ontology: Computer applications (pp. 231–243). New York, NY: Springer. Prinz, W., Aschersleben, G., & Koch, I. (2009). Cognition and action. In E. Morsella, J. A. Bargh, & P. M. Gollwitzer (Eds.), Oxford handbook of human action (pp. 33–70). New York, NY: Oxford University Press. Regev, M., Honey, U., & Hasson, U. (2013). Modality-selective and modality-invariant neural responses to spoken and written narratives. Journal of Neuroscience, 33, 15978–15988. Rodgers, J. L. (2010). The epistemology of mathematical and statistical modeling: A quiet methodological revolution. American Psychologist, 65, 1–12. doi:https://doi.org/10.1037/a0018326 Russell, B. (1922). An introduction to the Tractatus Logico-Philosophicus. In L. Wittgenstein, Tractatus logico-philosophicus. London, UK: Kegan Paul. Schwarz, N. (1999). Self-reports: How the questions shape the answers. American Psychologist, 54, 93–105. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin. Shore, L. M., Tetrick, L. E., Lynch, P., & Barksdale, K. (2006). Social and economic exchange: Construct development and validation. Journal of Applied Social Psychology, 36, 837–867. Slaney, K. L. (2017). Validating psychological constructs: Historical, philosophical, and practical dimensions. London, UK: Palgrave MacMillan. Slaney, K. L., & Racine, T. P. (2013). What’s in a name? Psychology’s ever evasive construct. New Ideas in Psychology, 31, 4–12. doi:https://doi.org/10.1016/j.newideapsych.2011.02.003 Smedslund, J. (1994). Nonempirical and empirical components in the hypotheses of 5 social-psychological experiments. Scandinavian Journal of Psychology, 35, 1–15. Stogdill, R. M. (1963). Manual for the leader behavior description questionnaire, form XII. Columbus, OH: Bureau of Business Research, Ohio State University. Triandis, H. C. (1980). Value, attitudes and interpersonal behavior. In M. M. Page (Ed.), Nebraska Symposium on Motivation, Beliefs, Attitudes and Values (pp. 159–295). Lincoln, NE: University of Nebraska. Tversky, B. (1993). Cognitive maps, cognitive collages, and spatial mental models. In A. U. Frank & I. Campari (Eds.), Spatial information theory: A theoretical basis for GIS (pp. 14–24). Berlin, Germany: Springer. Van Berkum, J. J. A., Koornneef, A. W., Otten, M., & Neuwland, M. S. (2007). Establishing reference in language comprehension: An electrophysiological perspective. Brain Research, 1146, 158–171. doi:https://doi.org/10.1016/j.brainres.2006.06.091 van der Linden, D., te Nijenhuis, J., & Bakker, A. B. (2010). The general factor of personality: A meta-analysis of Big Five intercorrelations and a criterion-related validity study. Journal of Research in Personality, 44, 315–327. doi:https://doi.org/10.1016/j.jrp.2010.03.003. Van Dyne, L., & LePine, J. A. (1998). Helping and voice extra-role behaviors: Evidence of construct and predictive validity. Academy of Management Journal, 41, 108–119. Van Knippenberg, D., & Sitkin, S. B. (2013). A critical assessment of charismatic–transformational leadership research: Back to the drawing board? Academy of Management Annals, 7, 1–60. doi:https://doi.org/10.1080/19416520.2013.759433 van Schuur, W. H. (2003). Mokken scale analysis: Between the Guttman scale and parametric item response theory. Political Analysis, 11, 139–163. doi:https://doi.org/10.1093/pan/mpg002 Wittgenstein, L. (1922). Tractatus logico-philosophicus. London, UK: Kegan Paul. Zimbardo, P. G., Ebbesen, E. B., & Maslach, C. (1977). Influencing attitudes and changing behavior (2nd ed.). Reading, MA: Addison Wesley.