Developing evidence for a validity argument for an English placement exam from multi-year test performance data
Tóm tắt
This study investigated the factor structure and factorial invariance of an English Placement Exam (EPE) from 1998 to 2011 to provide evidence for both the appropriateness of the test scores interpretations and for a validity argument. Test performance data collected from 38,632 freshmen non-English majors from a university in central Taiwan from 13 years (1998–2001, and 2003–2011) was examined using both exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). EFA was performed on 6 years of data (2006–2011) to establish a baseline structure, which was then further tested using CFA on data from all the years. Results from EFA supported a three-factor oblique (correlated) solution and CFA determined that a three-factor model as the best fit for the data. This model reflected the test structure posited by the test designers (grammar, reading, and listening sections) and remained factorially invariant and factorially distinct across all years, with insignificant variation in factor loadings and model fit indices. The study concluded that the factor structure of the EPE provided evidence for a construct validity argument for the placement exam based on multi-year performance data, thus supporting the inferences regarding test scores interpretations and the soundness of score comparisons made across years.
Tài liệu tham khảo
Abedi, J. (2002). Standardized achievement tests and English language learners: Psychometrics issues. Educational Assessment, 8, 231–257.
Bachman, L. F. (1982). The trait structure of cloze test scores. TESOL Quarterly, 16, 61–70.
Bachman, L. F., Davidson, F. G., Ryan, K., & Choi, I.-C. (1995). An investigation into the comparability of two tests of English as a foreign language: The Cambridge-TOEFL comparability study. Cambridge, England: UCLES.
Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice. Oxford, England: Oxford University Press.
Bachman, L. F., & Palmer, A. S. (1981). The construct validation of the FSI oral interview. Language Learning, 31, 67–86.
Bae, J., & Bachman, L. F. (1998). A latent variable approach to listening and reading: testing factorial invariance across two groups of children in the Korean/English Two-Way Immersion Program. Language Testing, 15, 380–414.
Blais, J. G., & Laurier, M. D. (1995). The dimensionality of a placement test from several analytical perspectives. Language Testing, 12, 72–98.
Boldt, R. F. (1998). Latent structure analysis of the Test of English as a Foreign Language. TOEFL Research Report 28. Princeton, NJ: Educational Testing Service.
Byrne, B. M. (1994). Structural equation modeling with EQS and EQS/Windows. Thousand Oaks, CA: Sage Publications.
Carr, N. (2000). A comparison of analytic and holistic rating scale types in the context of composition tests. Issues in Applied Linguistics, 11, 207–241.
Carr, N. (2006). The factor structure of test task characteristics and examinee performance. Language Testing, 2006(23), 269–289.
Farhady, H. (1983). On the plausibility of the unitary language proficiency factor. In J. W. Oller (Ed.), Issues in language testing research (pp. 11–28). Rowley, MA: Newbury House.
Hale, G. A., Rock, D. A., & Jirele, T. (1989). Confirmatory factor analysis of the TOEFL.TOEFL Research Report 32. Princeton, NJ: Educational Testing Service.
Kunnan, A. J. (1994). Modeling relationships among some test-taker characteristics and performance on EFL tests: an approach to construct validity. Language Testing, 11, 225–252.
Kunnan, A. J. (1998). Approaches to validation in language assessment. In A. Kunnan (Ed.), Validation in language assessment (pp. 1–8). Mahwah, NJ: Lawrence Erlbaum Associates.
Oller, J. W., & Hinofotis, F. (1980). Two mutually exclusive hypotheses about second language ability: Indivisible or partially divisible competence. In J. W. Oller & K. Perkins (Eds.), Research in language testing (pp. 13–23). Rowley, MA: Newbury House.
Oller, J. W. (1983). Evidence for a general language proficiency factor: An expectancy grammar. In J. W. Oller (Ed.), Issues in language testing research (pp. 3–10). Rowley, MA: Newbury House.
Oltman, P. K., Stricker, L. J., & Barrows, T. (1988). Native language, English proficiency and the structure of the TOEFL (TOEFL Research Report 27). Princeton, NJ: Educational Testing Service.
Pomplun, M., & Omar, M. (2003). Do minority representative reading passages provide factorially invariant scores for all students? Structural Equation Modeling, 10, 276–288.
Romhild, A. (2008). Investigating the factor structure of the ECPE across different proficiency levels. Spaan Fellow Working Papers in Second of Foreign Language Assessment, 6, 29–55.
Sawaki, Y., Stricker, S., & Oranje, A. (2008). Factor structure of the TOEFL Internet-Based Test (iBT): Exploration in a field trial sample. TOEFL iBT Research Report 4. Princeton, NJ: Educational Testing Service.
Shin, S. (2005). Did they take the same test? Examinee language proficiency and the structure of language tests. Language Testing, 22, 31–57.
Sims, J. (2006). The creation of a valid and reliable university proficiency exam. Tunghai Journal of Humanities, 47, 325–344.
Sims, J., & Liu, J. (2013). Two decades of changes in the English ability of freshmen at a university in Taiwan. Hwa Kang English Journal, 19, 23–51.
Song, M. Y. (2008). Do divisible subskills exist in second language (L2) comprehension? A structural equation modeling approach. Language Testing, 25, 435–464.
Stevens, J. (1992). Applied multivariate statistics for social science (2nd ed.). Hillsdale, NJ: Erlbaum.
Stricker, L. J. (2004). The performance of native speakers of English and ESL speakers on the computer-based TOEFL and GRE General Test. Language Testing, 21, 146–173.
Tomblin, J. B., & Zhang, X. (2007). The dimensionality of language ability in school-age children. Journal of Speech, Language, and Hearing Research, 49, 1193–1208.
Vollmer, H., & Sang, F. (1983). Competing hypotheses about second language ability. A plea for caution. In J. W. Oller (Ed.), Issues in language testing research (pp. 29–79). Rowley, MA: Newbury House.