Alleviating the Cold Start Problem in Adaptive Learning using Data-Driven Difficulty Estimates
Tóm tắt
An adaptive learning system offers a digital learning environment that adjusts itself to the individual learner and learning material. By refining its internal model of the learner and material over time, such a system continually improves its ability to present appropriate exercises that maximise learning gains. In many cases, there is an initial mismatch between the internal model and the learner’s actual performance on the presented items, causing a “cold start” during which the system is poorly adjusted to the situation. In this study, we implemented several strategies for mitigating this cold start problem in an adaptive fact learning system and experimentally tested their effect on learning performance. The strategies included predicting difficulty for individual learner-fact pairs, individual learners, individual facts, and the set of facts as a whole. We found that cold start mitigation improved learning outcomes, provided that there was sufficient variability in the difficulty of the study material. Informed individualised predictions allowed the system to schedule learners’ study time more effectively, leading to an increase in response accuracy during the learning session as well as improved retention of the studied items afterwards. Our findings show that addressing the cold start problem in adaptive learning systems can have a real impact on learning outcomes. We expect this to be particularly valuable in real-world educational settings with large individual differences between learners and highly diverse materials.
Tài liệu tham khảo
Anderson, J.R. (1981). Effects of prior knowledge on memory for new information. Memory & Cognition, 9(3), 237–246. https://doi.org/10.3758/BF03196958.
Anderson, J.R. (2007). How Can the Human Mind Occur in the Physical Universe? Oxford. United Kingdom: Oxford University Press.
Anderson, J.R., Corbett, A.T., Koedinger, K.R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. Journal of the Learning Sciences, 4(2), 167–207. https://doi.org/10.1207/s15327809jls0402_2.
Bobadilla, J., Ortega, F., Hernando, A., & Bernal J. (2012). A collaborative filtering approach to mitigate the new user cold start problem. Knowledge-Based Systems, 26, 225–238. https://doi.org/10.1016/j.knosys.2011.07.021.
Broers, N., & Busch, N. (2019). The effect of intrinsic image memorability on recollection and familiarity (Preprint). PsyArXiv. https://doi.org/10.31234/osf.io/hvzyw.
Bürkner, P.C. (2017). Brms: An R package for bayesian multilevel models using stan. Journal of Statistical Software, 80(1), 1–28. https://doi.org/10.18637/jss.v080.i01.
Celikkale, B., Erdem, A., & Erdem, E. (2013). Visual Attentionx-Driven Spatial Pooling for Image Memorability. In 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops (976–983). OR, USA: IEEE. https://doi.org/10.1109/CVPRW.2013.142.
Chen, S.-Y., Ankenmann, R.D., & Chang, H.-H. (2000). A comparison of item selection rules at the early stages of computerized adaptive testing. Applied Psychological Measurement, 24(3), 241–255. https://doi.org/10.1177/01466210022031705.
Corbett, A.T., Koedinger, K.R., & Anderson, J.R. (1997). Intelligent Tutoring Systems. In Helander, M.G., Landauer, T.K., & Prabhu, P.V. (Eds.) Handbook of human-computer interaction. 2nd edn. (pp. 849–874). Amsterdam: Elsevier.
de Leeuw, J.R. (2015). jsPsych: A javascript library for creating behavioral experiments in a web browser. Behavior Research Methods, 47(1), 1–12. https://doi.org/10.3758/s13428-014-0458-y.
Dempster, F.N. (1988). A case study in the failure to apply the results of psychological research. American Psychologist, 43(8), 627–634.
Eliot, C.R., Williams, K.A., & Woolf, B.P. (1996). An intelligent learning environment for advanced cardiac life support. In Proceedings of the AMIA Annual Fall Symposium (pp 7–11) Washington, DC, American Medical Informatics Association.
Gallistel, C.R. (2009). The importance of proving the null. Psychological Review, 116(2), 439–453. https://doi.org/10.1037/a0015251.
Genest, C., Weerahandi, S., & Zidek, J.V. (1984). Aggregating opinions through logarithmic pooling. Theory and Decision, 17(1), 61–70. https://doi.org/10.1007/BF00140056.
Gronau, Q.F., Singmann, H., & Wagenmakers, E.J. (2020). Bridgesampling: An R package for estimating normalizing constants. Journal of Statistical Software, 92(10), 1–29. https://doi.org/10.18637/jss.v092.i10.
Hamari, J., Shernoff, D.J., Rowe, E., Coller, B., Asbell-Clarke, J., & Edwards, T. (2016). Challenging games help students learn: An empirical study on engagement, flow and immersion in game-based learning. Computers in Human Behavior, 54, 170–179. https://doi.org/10.1016/j.chb.2015.07.045.
Jeffreys, H. (1961). Theory of probability, Third ed. Oxford: Oxford University Press.
Kennedy, P., Miele, D.B., & Metcalfe, J. (2014). The cognitive antecedents and motivational consequences of the feeling of being in the zone. Consciousness and Cognition, 30, 48–61. https://doi.org/10.1016/j.concog.2014.07.007.
Klinkenberg, S., Straatemeier, M., & van der Maas, H. (2011). Computer adaptive practice of Maths ability using a new item response model for on the fly ability and difficulty estimation. Computers & Education, 57(2), 1813–1824. https://doi.org/10.1016/j.compedu.2011.02.003.
Lika, B., Kolomvatsos, K., & Hadjiefthymiades, S. (2014). Facing the cold start problem in recommender systems. Expert Systems with Applications, 41(4), 2065–2073. https://doi.org/10.1016/j.eswa.2013.09.005.
Lops, P., de Gemmis, M., & Semeraro, G. (2011). Content-based Recommender systems: state of the art and trends. In Ricci, F., Rokach, L., Shapira, B., & Kantor, P.B. (Eds.) Recommender systems handbook (pp 73–105) Boston, MA, Springer US.
Madan, C.R. (2019). How well do different word properties explain item free recall? (Preprint). PsyArXiv. https://doi.org/10.31234/osf.io/kfh87.
Metcalfe, J., & Kornell, N. (2005). A region of proximal learning model of study time allocation. Journal of Memory and Language, 52(4), 463–477. https://doi.org/10.1016/j.jml.2004.12.001.
Mettler, E., Massey, C.M., & Kellman, P.J. (2016). A comparison of adaptive and fixed schedules of practice. Journal of Experimental Psychology: General, 145(7), 897–917. https://doi.org/10.1037/xge0000170.
Morey, R.D., & Rouder, J.N. (2018). BayesFactor: computation of bayes factors for common designs.
Murphy, K.P. (2007). Conjugate Bayesian Analysis of the Gaussian Distribution (Technical Report). Vancouver: BC:University of British Columbia.
Nedungadi, P., & Remya, M.S. (2014). Predicting students’ performance on intelligent tutoring system – Personalized clustered BKT (PC-BKT) model. In 2014 IEEE Frontiers in Education Conference (FIE) Proceedings 2014 Proceedings (pp 1–6). Madrid, Spain: IEEE. https://doi.org/10.1109/FIE.2014.7044200.
Nijboer, M. (2011). Optimal Fact Learning: Applying Presentation Scheduling to Realistic Conditions (Unpublished Master’s Thesis). Groningen: University of Groningen.
Pardos, Z.A., & Heffernan, N.T. (2010). Modeling Individualization in a Bayesian Networks Implementation of Knowledge Tracing. In Proceedings of the 18th International Conference on User Modeling, Adaptation and Personalization (pp 255–266), Big Island, Hawaii. https://doi.org/10.1007/978-3-642-13470-8-24.
Park, J.Y., Joo, S.H., Cornillie, F., van der Maas, H.L.J., & van den Noortgate, W. (2018). An explanatory item response theory method for alleviating the cold-start problem in adaptive learning environments. Behavior Research Methods, 51(2), 895–909. https://doi.org/10.3758/s13428-018-1166-9.
Pavlik, P.I., & Anderson, J.R. (2005). Practice and forgetting effects on vocabulary memory: an activation-based model of the spacing effect. Cognitive Science, 29(4), 559–586. https://doi.org/10.1207/s15516709cog000014.
Pavlik, P.I., & Anderson, J.R. (2008). Using a model to compute the optimal schedule of practice. Journal of Experimental Psychology: Applied, 14(2), 101–117. https://doi.org/10.1037/1076-898X.14.2.101.
Pliakos, K., Joo, S.H., Park, J.Y., Cornillie, F., Vens, C., & van den Noortgate, W. (2019). Integrating machine learning into item response theory for addressing the cold start problem in adaptive learning systems. Computers & Education, 137, 91–103. https://doi.org/10.1016/j.compedu.2019.04.009.
Popescul, A., Ungar, L.H., Pennock, D.M., & Lawrence, S. (2001). Probabilistic Models for Unified Collaborative and Content-Based Recommendation in Sparse-Data Environments. In Proceedings of the 17th Conference on Uncertainty in Artificial Intelligence (pp 437–444). San Francisco, CA.
R Core Team. (2018). R: a language and environment for statistical computing. r foundation for statistical computing. Vienna, Austria.
Reiser, B.J., Anderson, J.R., & Farrell, R.G. (1985). Dynamic Student Modelling in an Intelligent Tutor for LISP Programming. In Proceedings of the 9th international joint conference on artificial intelligence (pp 8–14). Los Angeles, CA.
Rouder, J.N. (2014). Optional stopping: No problem for Bayesians. Psychonomic Bulletin & Review, 21(2), 301–308. https://doi.org/10.3758/s13423-014-0595-4.
Rouder, J.N., & Morey, R.D. (2012). Default Bayes factors for model selection in regression. Multivariate Behavioral Research, 47(6), 877–903.
Schönbrodt, F.D., Wagenmakers, E.J., Zehetleitner, M., & Perugini, M. (2017). Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences. Psychological Methods, 22(2), 322–339. https://doi.org/10.1037/met0000061.
Sense, F., Behrens, F., Meijer, R.R., & van Rijn, H. (2016). An individual’s rate of forgetting is stable over time but differs across materials. Topics in Cognitive Science, 8(1), 305–321. https://doi.org/10.1111/tops.12183.
Sense, F., van der Velde, M., & van Rijn, H. (2018). Deploying a Model-based Adaptive Fact-Learning System in University Courses. In Proceedings of the 16th International Conference on Cognitive Modeling (pp 136–137), Madison, WI.
Shernoff, D.J., Csikszentmihalyi, M., Schneider, B., & Shernoff, E.S. (2003). Student Engagement in High School Classrooms from the Perspective of Flow Theory. In Applications of Flow in Human Development and Education (pp 19). https://doi.org/10.1007/978-94-017-9094-9-24. Dordrecht: Springer.
van den Broek, G., Segers, E., van Rijn, H., Takashima, A., & Verhoeven, L. (2019). Effects of elaborate feedback during practice tests: Costs and benefits of retrieval prompts. Journal of Experimental Psychology: Applied.
van den Broek, G., Takashima, A., Wiklund-Hörnqvist, C., Karlsson Wirebring, L., Segers, E., & Verhoeven, L. (2016). Neurocognitive mechanisms of the “testing effect”: a review. Trends in Neuroscience and Education, 5(2), 52–66. https://doi.org/10.1016/j.tine.2016.05.001.
van Rijn, H., van Maanen, L., & van Woudenberg, M. (2009). Passing the test: Improving learning gains by balancing spacing and testing effects. In Proceedings of the 9th International Conference on Cognitive Modeling (pp 110–115). Manchester, United Kingdom.
VanLehn, K. (2006). The Behavior of Tutoring Systems. International Journal of Artificial Intelligence in Education, 16, 227–265.
Wagenmakers, E.-J., Lodewyckx, T. , Kuriyal, H., & Grasman, R. (2010). Bayesian hypothesis testing for psychologists: A tutorial on the savage–dickey method. Cognitive Psychology, 60(3), 158–189. https://doi.org/10.1016/j.cogpsych.2009.12.001.
Wauters, K., Desmet, P., & van den Noortgate, W. (2010). Adaptive item-based learning environments based on the item response theory: Possibilities and challenges: Adaptive ITSs based on IRT. Journal of Computer Assisted Learning, 26(6), 549–562. https://doi.org/10.1111/j.1365-2729.2010.00368.x.
Zhou, P., Sense, F., van Rijn, H., & Stocco, A. (2020). Reflections of idiographic long-term memory characteristics in resting-state neuroimaging data (Preprint). bioRxiv. https://doi.org/10.1101/2020.04.18.047662.