Assessing Japanese junior high school students’ English achievement through computer-based testing in the classroom: a case of integrated reading-into-writing continuous task
Tóm tắt
This paper describes the exploratory case and initial evaluation of the computer-based testing (CBT) prototype. The advantage of CBT over paper-based testing (PBT) is that it allows us to control the order of questions and provides test takers with continuous tasks capturing their thought processes. Additionally, their response process data such as response time (RT) can be obtained. Taking advantage of these, we created a CBT prototype in the classroom for Japanese junior high school students. A CBT model was created to assess integrated reading and writing ability and was administered to 32 junior high school students. Their process achievement and the relation between the process response and writing quality were analyzed. Students’ RT for each screen was analyzed using hierarchical cluster analysis. We identified not only students facing difficulties at each stage of a series of thought processes but also five clusters that include students spending too much time reading source texts or organizing their ideas. We suggest how CBT can be developed to identify students with difficulties and applied to teaching. CBT has the possibility of detecting students who are able to complete the language performance task by controlling the order of answers, asking questions sequentially, and obtaining RT effectively.
Tài liệu tham khảo
Barkaoui, K. (2014). Examining the impact of L2 proficiency and keyboarding skills on scores on TOEFL-iBT writing tasks. Language Testing, 31(2), 241–259. https://doi.org/10.1177/0265532213509810
Bennett, R., Zhang, M., Sinharay, S., Guo, H., & Deane, P. (2022). Are there distinctive profiles in examinee essay-writing processes? Educational Measurement: Issues and Practice, 41(2), 55–69. https://doi.org/10.1111/emip.12469
Brunfaut, T., Harding, L., & Batty, A. (2018). Going online: the effect of mode of delivery on performances and perceptions on an English L2 writing test suite. Assessing Writing, 36, 3–18. https://doi.org/10.1016/j.asw.2018.02.003
Chan, S. (2017). Using keystroke logging to understand writers’ processes on a reading-into-writing test. Language Testing in Asia, 7, 10. https://doi.org/10.1186/s40468-017-0040-5
Chan, S., Inoue, C., & Taylor, L. (2015). Developing rubrics to assess the reading-into-writing skills: A case study. Assessing Writing, 26, 20–37. https://doi.org/10.1016/j.asw.2015.07.004
Chan, S., Bax, S., & Weir, C. (2018). Researching the comparability of paper-based and computer-based delivery in a high-stakes writing test. Assessing Writing, 36, 32–48. https://doi.org/10.1016/j.asw.2018.03.008
Douglas, D., & Hegelheimer, V. (2007). Assessing language using computer technology. Annual Review of Applied Linguistics, 27, 115–132. https://doi.org/10.1017/S0267190508070062
Ercikan, K., & Pellegrino, J. (Eds.) (2018). Validation of score meaning for the next generations of assessments: The use of response processes. Routledge
Gong, T., Shuai, L., Arslan, B., & Jiang, Y. (2020). Process based analysis on scientific inquiry tasks using large-scale national assessment dataset. Proceedings of the 13th international conference on Educational Data Mining (EDM 2020), 417–423. https://educationaldatamining.org/files/conferences/EDM2020/papers/paper_7.pdf
Jamieson, J. (2005). Trends in computer-based second language assessment. Annual Review of Applied Linguistics, 25, 228–242. https://doi.org/10.1017/S0267190505000127
Kitazawa, T., & Shirouzu, H. (2020). CBT niyoru tashisentakushikimondai no kaiketsupurosesu no kaimei: Daigakunyuushi Senta shikenmondai no kokugokisyutsumondai wo katsuyou shite [Clarifying solving processes of multiple-choice problems by CBT: Using past Japanese language tests of the National Center University Entrance Examination]. University Entrance Examination Research Journal, 30, 52–58.
Knoch, U., & Sitajalabhorn, W. (2013). A closer look at integrated writing tasks: towards a more focussed definition for assessment purposes. Assessing Writing, 18(4), 300–308. https://doi.org/10.1016/j.asw.2013.09.003
Lee, Y. H., Hao, J., Man, K., & Ou, L. (2019). How do test takers interact with simulation-based tasks? A response-time perspective. Frontiers in Psychology, 10, 906. https://doi.org/10.3389/fpsyg.2019.00906
Masukawa, H., Shirouzu, H., Saito, M., Iikubo, S., & Amano, T. (2021). Development of CBT reading items to elicit “aggressive reading”: Using language test of the university of Tokyo entrance examination. Japanese Journal for Research on Testing, 17(1), 25–44. https://doi.org/10.24690/jart.17.1_25
Matsuura, N. (2021). In Y, Ushiro., & M, Kashiba. (Eds.). Shin kyosyoku katei ensyu: Vol. 18. Chuto eigoka kyouiku [Middle English Education](pp. 10–13). Kyodo Shuppan.
MEXT. (2017). Chugakkou Gakusyuu Shidou Youryou (Heisei 29 nendo kokuji) kaisetsu gaikoguko hen [The Explanation of The Course of Study for Junior High School (public notification in 29th year of Heisei period) Foreign Language Version].
MEXT. (2019). Heisei 31 nendo (Reiwa gan nendo) zenkoku gakuryoku gakusyu jyokyo chosa hokokusyo Chugakko eigo [In 31th year of Heisei period (In 1st year of Reiwa period) The nationwide academic achievement test report Junior high school English].
National Center for university entrance examinations (2021). Daikibo Nyugakusya Senbatsu Niokeru CBT Katsuyou no Kanousei Nitsuite (Hokoku) [The Possibility of Using CBT in Large-Scale Admissions (Report)] https://www.dnc.ac.jp/albums/abm.php?f=abm00040361.pdf&n=%E3%80%90%E5%90%88%E4%BD%93%E7%89%88%E3%80%91CBT%E5%A0%B1%E5%91%8A%E6%9B%B8.pdf
Nishigori, D., Yamaguchi, A., Matsutaka, K., Osada, S., Sakaguchi, K., Fukui, T., Takamori, Y., Sonoda, Y., & Kodama, H. (2017). Dejitarugijyutsu wo katsuyou shita taburetto nyushi no kaihatsu: Tamenteki, Sougoutekihyouka ni muketa gijyutsuteki kentou [Development of examination using tablet device: technical considerations for multi-source and comprehensive assessment]. University Entrance Examination Research Journal, 27, 63–69.
Plakans, L. (2008). Comparing composing processes in writing-only and reading-to-write test tasks. Assessing Writing, 13(2), 111–129. https://doi.org/10.1016/j.asw.2008.07.001
Sahin, F., & Colvin, K. F. (2020). Enhancing response time thresholds with response behaviors for detecting disengaged examinees. Large-scale Assessments in Education, 8, 5. https://doi.org/10.1186/s40536-020-00082-1
Talebinamvar, M., & Zarrabi, F. (2022). Clustering students’ writing behaviors using keystroke logging: a learning analytic approach in EFL writing. Language Testing in Asia, 12, 6. https://doi.org/10.1186/s40468-021-00150-5
The Japan Association for research on testing (2007). Test Standard: Nihon no test no shourai ni mukete [Test Standards: Toward the future of testing in Japan], Kaneko Shobo.
Ukon, S., Kobayashi, M., Nakamura, Y., Okamoto, E., Yuji, S., Terasaki, S., Yasuda, J., & Yasuno, F. (2019). Taburetto tanmatsu wo mochiita eizou ya doutekiobujekuto wo fukumu CBT butsuri mondai no kaihatsu [Developing of physics problems with movies and dynamic objects for CBT using tablet computer]. Proceedings of the 43rd annual meeting of Japan Society for Science Education, 195–198. https://doi.org/10.14935/jssep.43.0_195
Xu, C., & Ding, Y. (2014). An exploratory study of pauses in computer-assisted EFL writing. Language Learning & Technology, 18(3), 80–96. 10125/44385.
Yamashita, T. (2017). Computer-based-testing (CBT) mondai no shisaku [A prototype of Computer-Based-Testing (CBT)]. Chemistry in Education, 65(7), 334–337. https://doi.org/10.20665/kakyoshi.65.7_334
Yang, H. (2012). A comparative study of composing processes in reading- and graph-based writing tasks. Language Testing in Asia, 2(3), 33–52. https://doi.org/10.1186/2229-0443-2-3-33