Investigating the factor structure of the Test of English for Academic Purposes (TEAP) and its relation to test takers’ perceived test task value
Tóm tắt
This study investigated the scoring and criterion-related validity of the TEAP, a newly developed Test of English for Academic Purposes. In this study, scoring validity was examined by investigating the factor structure, while criterion-related validity was examined by first investigating the longitudinal change of test takers’ perceived test task value toward the measured construct and then investigating the relationship of test takers’ perceived value to the factor structure of the TEAP. Confirmatory item-level factor analysis was conducted using the data obtained from 2217 first-year university students comparing four models (unitary, correlated, receptive-productive, higher order). Additional confirmatory factor analyses were conducted to first investigate the longitudinal change of perceived value toward the measured construct and then to investigate the relationship of test takers’ perceived value of the construct measured by the test to the factor structure of the TEAP. The results show that the higher-order model was the best-fitting model. This confirmed a previous small-scale study suggesting the generalizability of the test’s factor structure. Furthermore, it was found that test takers’ perceived values measured at the start of university positively affected the values measured about 6 months later. In addition, perceived values measured both at the start of university study and about 6 months later positively correlated with the higher-order factor of the test. The results provide further support of the scoring validity of the test. In addition, a positive relationship between the higher-order factor of TEAP and the factors of perceived values provide evidence of the usefulness of test takers’ perception to further support the criterion-related validity of the test.
Tài liệu tham khảo
Allen, D. (2020). Proposing change in university entrance examinations: A tale of two metaphors. TEVAL - Shiken: A Journal of Language Testing and Evaluation in Japan, 24(2), 23–38.
Bachman, L., & Palmer, A. (1981). The construct validation of the FSI oral interview. Language Learning, 31(1), 67–77.
Benesse. (2022). GTEC CBT. http://www.benesse-gtec.com/cbt/en. Accessed 11 May 2022.
British Council, IDP, IELTS Australia, & Cambridge English Language Assessment (2022). IELTS. http://www.ielts.org/. Accessed 11 May 2022.
Byrne, B. (2012). Structural equation modeling with Mplus. New York: Routledge.
Chalhoub-Deville, M. (2016). Validity theory: Reform policies, accountabilities testing, and consequences. Language Testing, 33(4), 453–472. https://doi.org/10.1177/0265532215593312.
Council of Europe (2001). Common European Framework of Reference for Languages: Learning, teaching, assessment. Retrieved May 2022 from https://rm.coe.int/1680459f97
Dornyei, Z. (2005). The Psychology of the Language Learner. New Jersey: Routledge
Eccles, J. & Wigfield, A. (2002). Motivational Beliefs, Values, and Goals. Annual Review Psychology, 53, 109–132.
Educational Testing Service. (2010). Linking TOEFL iBT Scores to IELTS scores. A Research Report. Princeton, NJ: ETS.
Educational Testing Service. (2022). About the TOEFL iBT® test. https://www.ets.org/toefl/ibt/about. Accessed 11 May 2022.
Eiken Foundation of Japan. (2022a). EIKEN tests. http://www.eiken.or.jp/eiken/en/eiken-tests/. Accessed 11 May 2022.
Eiken Foundation of Japan. (2022b). TEAP kenkyu report [TEAP research reports]. http://www.eiken.or.jp/teap/group/report.html. Accessed 11 May 2022.
Green, A. (2014). The Test of English for Academic Purposes (TEAP) impact study: Report 1—Preliminary questionnaires to Japanese high school students and teachers. http://www.eiken.or.jp/teap/group/pdf/teap_washback_study.pdf. Accessed 11 May 2022.
Gu, L. (2015). Language ability of young English language learners: Definition, configuration, and implications. Language Testing, 32(1), 21–38. https://doi.org/10.1177/0265532214542670.
Horwitz, E. (1988). The Beliefs about Language Learning of Beginning University Foreign Language Students. The Modern Language Journal, 72(3), 283–294.
In’nami, Y., & Koizumi, R. (2011). Factor structure of the revised TOEIC test: A multiple-sample analysis. Language Testing, 29(1), 131–152. https://doi.org/10.1177/0265532211413444.
In’nami, Y., Koizumi, R., & Nakamura, K. (2016). Factor structure of the Test of English for Academic Purposes (TEAP) test in relation to the TOEFL iBT test. Language Testing in Asia, 6(3), 1–23. https://doi.org/10.1186/s40468-016-0025-9.
Kamiya, N. (2017). Can the National Center Test in Japan be replaced by commercially available private English tests of four skills? In the case of TOEFL Junior Comprehensive. Language Testing in Asia, 7(15), 1–22. https://doi.org/10.1186/s40468-017-0046-z.
Kuramoto, N., & Koizumi, R. (2016). Current issues in large-scale educational assessment in Japan: Focus on national assessment of academic ability and university entrance examinations. Assessment in Education: Principles, Policy & Practice. https://doi.org/10.1080/0969594X.2016.1225667.
Li, C. (2021). Understanding EAP learners’ beliefs about language learning from a socio-cultural perspective. Singapore: Springer.
Ministry of Education, Culture, Sports, Science and Technology. (2020). Daigaku Nyushi Seido no Genjou to Koudai Setsuzoku Kaikaku no Keii ni tsuite [On the history of the upper secondary school-university articulation reform and current status of college entrance exam, January 15, 2020]. https://www.mext.go.jp/content/20200116-mxt_daigakuc02-000004136_5.pdf. Accessed 11 May 2022.
Ministry of Education,Culture, Sports, Science and Technology (2015). Seito no eiryoku koujou puran [The plan for improving students’ English proficiency]. Retrieved June 28, 2022 from https://www.mext.go.jp/a_menu/kokusai/gaikokugo/__icsFiles/afieldfile/2015/07/21/1358906_01_1.pdf
Muthén, B. O. (2004). Mplus Technical Appendices. https://www.statmodel.com/download/techappen.pdf. Accessed 11 May 2022.
Muthén, L. K., & Muthén, B. O. (1998–2022). Mplus [computer software]. Los Angeles: Muthén & Muthén.
Nakamura, K. (2014). Examination of possible consequences of a new test within the context of university entrance exam reform in Japan. Paper presented at the 36th Language Testing Research Colloquium, VU University Amsterdam, the Netherlands. http://www.eiken.or.jp/teap/group/pdf/teap_ltrcpresentation20140620.pdf. Accessed 11 May 2022.
Nakatsuhara, F. (2014). A research report on the development of the Test of English for Academic Purposes (TEAP) speaking test for Japanese university entrants—Study 1 & study 2. http://www.eiken.or.jp/teap/group/pdf/teap_speaking_report1.pdf. Accessed 11 May 2022.
Nakatsuhara, F., Joyce, D., & Fouts, T. (2014). A research report on the development of the Test of English for Academic Purposes (TEAP) speaking test for Japanese university entrants—Study 3 & study 4. http://www.eiken.or.jp/teap/group/pdf/teap_speaking_report2.pdf. Accessed 11 May 2022.
National Center for University Entrance Examinations. (2017). Outline of the National Center for University Entrance Examinations. www.dnc.ac.jp/albums/abm00033004.pdf (dnc.ac.jp). Accessed 11 May 2022.
O’Sullivan, B., & Weir, C. (2011). Test development and validation. In B. O’Sullivan (Ed.), Language testing: theories and practices, (pp. 13–32). Oxford: Palgrave.
Oller Jr., J. W. (1980). Language testing research (1979–1980). Annual Review of Applied Linguistics, 1, 124–150.
Peacock, M. (2001). Pre-service ESL teachers’ beliefs about second language learning: A longitudinal study. System, 29, 177–195.
Powers, D.E., & Powers, A. (2015). The incremental contribution of TOEIC® Listening, Reading, Speaking, and Writing tests to predicting performance on real-life English language tasks. Language Testing, 32(2), 151–167. https://doi.org/10.1177/0265532214551855
Ross, S. (1998). Self-assessment in second language testing: a meta-analysis and analysis of experiential factors. Language Testing, 15(1), 1–20.
Runnels, J. (2016). Self-assessment accuracy: correlations between Japanese English learners’ self-assessment on the CEFR-Japan’s Can do statements and scores on the TOEIC. Taiwan Journal of TESOL, 13(1), 105–137.
Sasaki, M. (1993). Relationships among second language proficiency, foreign language aptitude, and intelligence: A structural equation modeling approach. Language Learning, 43(3), 313–344.
Sasaki, M. (2008). The 150-year history of English language assessment in Japanese education. Language Testing, 25(1), 63–83. https://doi.org/10.1177/0265532207083745.
Sawaki, Y., & Nissan, S. (2009). Criterion-related validity of the TOEFL iBT listening section (TOEFL iBT Research Report). Pronceton: ETS.
Sawaki, Y., Stricker, L., & Oranje, H. A. (2009). Factor structure of the TOEFL Internet-based test. Language Testing, 25(1), 53–0. https://doi.org/10.1177/0265532208097335
Taylor, L. (2014). A report on the review of test specifications for the reading and listening papers of the Test of English for Academic Purposes (TEAP) for Japanese university entrants. http://www.eiken.or.jp/teap/group/pdf/teap_rlspecreview_report.pdf. Accessed 11 May 2022.
University of Cambridge Local Examinations Syndicate. (2022). Cambridge English exams. http://www.cambridgeenglish.org/exams/. Accessed 11 May 2022.
Weir, C. (2014). A research report on the development of the Test of English for Academic Purposes (TEAP) writing test for Japanese university entrants. http://www.eiken.or.jp/teap/group/pdf/teap_writing_report.pdf. Accessed 11 May 2022.
Xie, Q. (2011). Is test taker perception of assessment related to construct validity? International Journal of Testing, 11(4), 324–348. https://doi.org/10.1080/15305058.2011.589018.
Xie, Q. (2015). Do component weighting and testing method affect time management and approaches to test preparation? A study on the washback mechanism. System, 50, 56–68. https://doi.org/10.1016/j.system.2015.03.002.
Xie, Q., & Andrews, S. (2013). Do test design and uses influence test preparation? Testing a model of washback with structural equation modeling. Language Testing, 30(1), 49–70. https://doi.org/10.1177/0265532212442634.