Assessment of factual recall and higher-order cognitive domains in an open-book medical school examination
Tóm tắt
Open-book examinations (OBEs) will likely become increasingly important assessment tools. We investigated how access to open-book resources affected questions testing factual recall, which might be easy to look-up, versus questions testing higher-order cognitive domains. Few studies have investigated OBEs using modern Internet resources or as summative assessments. We compared performance on an examination conducted as a traditional closed-book exam (CBE) in 2019 (N = 320) and a remote OBE with free access to Internet resources in 2020 (N = 337) due to COVID-19. This summative, end-of-year assessment focused on basic science for second-year medical students. We categorized questions by Bloom’s taxonomy (‘Remember’, versus ‘Understand/Apply’). We predicted higher performance on the OBE, driven by higher performance on ‘Remember’ questions. We used an item-centric analysis by using performance per item over all examinees as the outcome variable in logistic regression, with terms ‘Open-Book, ‘Bloom Category’ and their interaction. Performance was higher on OBE questions than CBE questions (OR 2.2, 95% CI: 2.14–2.39), and higher on ‘Remember’ than ‘Understand/Apply’ questions (OR 1.13, 95% CI: 1.09–1.19). The difference in performance between ‘Remember’ and ‘Understand/Apply’ questions was greater in the OBE than the CBE (‘Open-Book’ * ‘Bloom Category’ interaction: OR 1.2, 95% CI: 1.19–1.37). Access to open-book resources had a greater effect on performance on factual recall questions than higher-order questions, though performance was higher in the OBE overall. OBE design must consider how searching for information affects performance, particularly on questions measuring different domains of knowledge.
Tài liệu tham khảo
Agarwal, P. K., & Roediger, H. L. (2011). Expectancy of an open-book test decreases performance on a delayed closed-book test. Memory, 19(8), 836–852. https://doi.org/10.1080/09658211.2011.613840
Anderson, L. W., Krathwohl, D. R., & Bloom, B. S. (2001). A taxonomy for learning, teaching, and assessing: a revision of Bloom’s taxonomy of educational objectives. http://books.google.com/books?id=JPkXAQAAMAAJ&pgis=1. Accessed 4 January 2021
Baillie, C., & Toohey, S. (1997). The “power test”: Its impact on student learning in a materials science course for engineering students. Assessment and Evaluation in Higher Education, 22(1), 33–48. https://doi.org/10.1080/0260293970220103
Barsky, E., & Bar-Ilan, J. (2012). The impact of task phrasing on the choice of search keywords and on the search process and success. Journal of the American Society for Information Science and Technology, 63(10), 1987–2005. https://doi.org/10.1002/asi.22654
Bell, D. J., & Ruthven, L. (2004). Searcher’s assessments of task complexity for web searching. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 2997). https://doi.org/10.1007/978-3-540-24752-4_5
Betts, L. R., Elder, T. J., Hartley, J., & Trueman, M. (2009). Does correction for guessing reduce students’ performance on multiple-choice examinations? yes? no? sometimes? Assessment and Evaluation in Higher Education, 34(1), 1–15. https://doi.org/10.1080/02602930701773091
Bloom, B. S. (1965). Taxonomy of Educational Objectives: Cognitive domain (Vol. 1). New York: McKay, 0–111. http://nancybroz.com/nancybroz/Literacy_I_files/Bloom Intro.doc. Accessed 8 September 2020
Boniface, D. (1985). Candidates’ use of notes and textbooks during an open-book examination. Educational Research, 27(3), 201–209. https://doi.org/10.1080/0013188850270307
Broyles, I. L., Cyr, P. R., & Korsen, N. (2005). Open book tests: Assessment of academic learning in clerkships. Medical Teacher, 27(5), 456–462. https://doi.org/10.1080/01421590500097075
Byström, K., & Järvelin, K. (1995). Task complexity affects information seeking and use. Information Processing and Management, 31(2), 191–213. https://doi.org/10.1016/0306-4573(95)80035-R
Campbell, D. J. (1988). Task complexity: A review and analysis. Academy of Management Review, 13(1), 40–52. https://doi.org/10.5465/amr.1988.4306775
Chevalier, A., Dommes, A., & Marquié, J. C. (2011). Information searching on the web: The cognitive difficulties experienced by older users in modifying unsuccessful information searches. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 6781 LNAI, pp. 225–232). Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-21741-8_25
Craig, C., Kasana, N., & Modi, A. (2020). Virtual OSCE delivery: The way of the future? Medical Education, medu.14286. https://doi.org/10.1111/medu.14286
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika (Vol. 16). https://doi.org/10.1007/BF02310555
Cunnington, J. P. W., Norman, G. R., Blake, J. M., Dauphinee, W. D., & Blackmore, D. E. (1996). Applying learning taxonomies to test items: Is a fact an artifact? Academic Medicine, 71(10), s31–s33. https://doi.org/10.1007/978-94-011-4886-3_40
Dickson, K. L., & Miller, M. D. (2005). Authorized crib cards do not improve exam performance. Teaching of Psychology, 32(4), 230–233. https://doi.org/10.1207/s15328023top3204_6
Dinet, J., Chevalier, A., & Tricot, A. (2012). Information search activity: An overview. Revue Europeene De Psychologie Appliquee, 62(2), 49–62. https://doi.org/10.1016/j.erap.2012.03.004
Dommes, A., & Chevalier, A. (2011). The role of cognitive flexibility and vocabulary abilities of younger and older users in searching for information on the web. Article in Applied Cognitive Psychology, 25(5), 717–726. https://doi.org/10.1002/acp.1743
Durning, S. J., Dong, T., Ratcliffe, T., Schuwirth, L., Artino, A. R., Boulet, J. R., & Eva, K. (2016, April 1). Comparing open-book and closed-book examinations: A systematic review. Academic Medicine. Lippincott Williams and Wilkins. https://doi.org/10.1097/ACM.0000000000000977
Eva, K. W., & Anderson, M. B. (2020). Medical education adaptations: Really good stuff for educational transition during a pandemic. Medical Education, 54(6), 494–494. https://doi.org/10.1111/medu.14172
Ford, N., Miller, D., & Moss, N. (2005). Web search strategies and human individual differences: Cognitive and demographic factors, internet attitudes, and approaches. Journal of the American Society for Information Science and Technology, 56(7), 741–756. https://doi.org/10.1002/asi.20168
Fuller, R., Joynes, V., Cooper, J., Boursicot, K., & Roberts, T. (2020). Could COVID-19 be our ‘There is no alternative’ (TINA) opportunity to enhance assessment? Medical Teacher, 42(7), 781–786. https://doi.org/10.1080/0142159X.2020.1779206
Furst, E. J. (1981). Bloom’s taxonomy of educational objectives for the cognitive domain: Philosophical and educational issues. Review of Educational Research, 51(4), 441–453. https://doi.org/10.3102/00346543051004441
Gharib, A., Phillips, W., & Mathew, N. (2012). Cheat Sheet or Open-Book? A Comparison of the Effects of Exam Types on Performance, Retention, and Anxiety *. Psychology Research (Vol. 2).
Gierl, M. J. (1997). Comparing cognitive representations of test developers and students on a mathematics test with bloom’s taxonomy. Journal of Educational Research, 91(1), 26–32. https://doi.org/10.1080/00220679709597517
Goethals, G. R. (1987). Theories of group behavior: commentary. Theories of Group Behavior. https://doi.org/10.1007/978-1-4612-4634-3_10
Haladyna, T. M., & Rodriguez, M. C. (2013). Developing and validating test items. Developing and Validating Test Items. https://doi.org/10.4324/9780203850381
Hamamoto Filho, P. T., Silva, E., Ribeiro, Z. M. T., de Hafner, M., & L. M. B., Cecilio-Fernandes, D., & Bicudo, A. M. . (2020). Relationships between Bloom’s taxonomy, judges’ estimation of item difficulty and psychometric properties of items from a progress test: A prospective observational study. Sao Paulo Medical Journal, 138(1), 33–39. https://doi.org/10.1590/1516-3180.2019.0459.r1.19112019
Hancock, G. R. (1994). Cognitive complexity and the comparability of multiple-choice and constructed-response test formats. Journal of Experimental Education, 62(2), 143–157. https://doi.org/10.1080/00220973.1994.9943836
Hannon, P., Lappe, K., Griffin, C., Roussel, D., & Colbert-Getz, J. (2020). An objective structured clinical examination: From examination room to Zoom breakout room. Medical Education, 54(9), 861–861. https://doi.org/10.1111/medu.14241
Hargittai, E. (2010). Digital Na(t)ives? Variation in internet skills and uses among members of the “Net Generation.” Sociological Inquiry, 80(1), 92–113. https://doi.org/10.1111/j.1475-682X.2009.00317.x
Hargittai, E., & Hinnant, A. (2008). Digital inequality. Communication Research, 35(5), 602–621. https://doi.org/10.1177/0093650208321782
Heijne-Penninga, M., Kuks, J. B. M., Hofman, W. H. A., & Cohen-Schotanus, J. (2008). Influence of open- and closed-book tests on medical students’ learning approaches. Medical Education, 42(10), 967–974. https://doi.org/10.1111/j.1365-2923.2008.03125.x
Kane, M. (2006). Validation, (In: Brennan RL: editor. Educational measurement. 4th ed.). https://eduq.info/xmlui/handle/11515/34503. Accessed 14 June 2021
Kass, R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90(430), 773–795. https://doi.org/10.1080/01621459.1995.10476572
Krarup, N., Naeraa, N., & Olsen, C. (1974). Open-book tests in a university course. Higher Education, 3(2), 157–164. https://doi.org/10.1007/BF00143789
Larwin, K. H., Gorman, J., & Larwin, D. A. (2013). Assessing the impact of testing aids on post-secondary student performance: A meta-analytic investigation. Educational Psychology Review, 25(3), 429–443. https://doi.org/10.1007/s10648-013-9227-1
Lizzio, A., Wilson, K., & Simons, R. (2002). University students’ perceptions of the learning environment and academic outcomes: Implications for theory and practice. Studies in Higher Education, 27(1), 27–52. https://doi.org/10.1080/03075070120099359
Martinez, M. E. (1999). Cognition and the question of test item format. Educational Psychologist, 34(4), 207–218. https://doi.org/10.1207/s15326985ep3404_2
Marzano, R. J., & Kendall, J. S. (2007). The new taxonomy of educational objectives. The new taxonomy of educational objectives 2nd Ed (Vol. 2nd). http://www.amazon.com/New-Taxonomy-Educational-Objectives/dp/1412936292. Accessed 4 January 2021
Messick, S. (1984). The psychology of educational measurement. ETS Research Report Series, 1984(1), i–55. https://doi.org/10.1002/j.2330-8516.1984.tb00046.x
Momsen, J. L., Long, T. M., Wyse, S. A., & Ebert-May, D. (2010). Just the facts? introductory undergraduate biology courses focus on low-level cognitive skills. CBE Life Sciences Education, 9(4), 435–440. https://doi.org/10.1187/cbe.10-01-0001
Mooney, C. J., Peyre, S. E., Clark, N. S., & Nofziger, A. C. (2020). Rapid transition to online assessment: Practical steps and unanticipated advantages. Medical Education, 54(9), 857–858. https://doi.org/10.1111/medu.14225
Moore, B. R., & Jensen, P. A. (2007). Do Open-Book Exams Impede Long-Term Learning in Introductory Biology Courses ? Journal of College Science Teaching, 46–50. https://eric.ed.gov/?id=EJ769010. Accessed 8 January 2021
Robinson, L., Cotten, S. R., Ono, H., Quan-Haase, A., Mesch, G., Chen, W., et al. (2015). Digital inequalities and why they matter. Information Communication and Society, 18(5), 569–582. https://doi.org/10.1080/1369118X.2015.1012532
Sam, A. H., Reid, M. D., & Amin, A. (2020). High-stakes, remote-access, open-book examinations. Medical Education, 54(8), 767–768. https://doi.org/10.1111/medu.14247
Sanchiz, M., Chin, J., Chevalier, A., Fu, W. T., Amadieu, F., & He, J. (2017). Searching for information on the web: Impact of cognitive aging, prior domain knowledge and complexity of the search problems. Information Processing and Management, 53(1), 281–294. https://doi.org/10.1016/j.ipm.2016.09.003
Schwartzstein, R. M., & Roberts, D. H. (2017). Saying goodbye to lectures in medical school—Paradigm shift or passing fad? New England Journal of Medicine, 377(7), 605–607. https://doi.org/10.1056/NEJMp1706474
Tamblyn, R., Abrahamowicz, M., Dauphinee, D., Wenghofer, E., Jacques, A., Klass, D., et al. (2007). Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities. Journal of the American Medical Association, 298(9), 993–1001. https://doi.org/10.1001/jama.298.9.993
Theophilides, C., & Dionysiou, O. (1996). The major functions of the open-book examination at the university level: A factor analytic study. Studies in Educational Evaluation, 22(2), 157–170. https://doi.org/10.1016/0191-491X(96)00009-0
Trigwell, K., & Prosser, M. (1991). Improving the quality of student learning: The influence of learning context and student approaches to learning on learning outcomes. Higher Education, 22(3), 251–266. https://doi.org/10.1007/BF00132290
Ward, A. F. (2013). Supernormal: How the internet is changing our memories and our minds. Psychological Inquiry, 24(4), 341–348. https://doi.org/10.1080/1047840X.2013.850148
Zagury-Orly, I., & Durning, S. J. (2020). Assessing open-book examination in medical education: The time is now. Medical Teacher. Taylor and Francis Ltd. https://doi.org/10.1080/0142159X.2020.1811214