Gamification suffers from the novelty effect but benefits from the familiarization effect: Findings from a longitudinal study
Tóm tắt
There are many claims that gamification (i.e., using game elements outside games) impact decreases over time (i.e., the novelty effect). Most studies analyzing this effect focused on extrinsic game elements, while fictional and collaborative competition have been recently recommended. Additionally, to the best of our knowledge, no long-term research has been carried out with STEM learners from introductory programming courses (CS1), a context that demands encouraging practice and mitigating motivation throughout the semester. Therefore, the main goal of this work is to better understand how the impact of a gamification design, featuring fictional and competitive-collaborative elements, changes over a 14-week period of time, when applied to CS1 courses taken by STEM students (N = 756). In an ecological setting, we followed a 2x7 quasi-experimental design, where Brazilian STEM students completed assignments in either a gamified or non-gamified version of the same system, which provided the measures (number of attempts, usage time, and system access) to assess user behavior at seven points in time. Results indicate changes in gamification’s impact that appear to follow a U-shaped pattern. Supporting the novelty effect, the gamification’s effect started to decrease after four weeks, decrease that lasted between two to six weeks. Interestingly, the gamification’s impact shifted to an uptrend between six and 10 weeks after the start of the intervention, partially recovering its contribution naturally. Thus, we found empirical evidence supporting that gamification likely suffers from the novelty effect, but also benefits from the familiarization effect, which contributes to an overall positive impact on students. These findings may provide some guidelines to inform practitioners about how long the initial contributions of gamification last, and how long they take to recover after some reduction in benefits. It can also help researchers to realize when to apply/evaluate interventions that use gamification by taking into consideration the novelty effect and, thereby, better understand the real impact of gamification on students’ behavior in the long run.
Tài liệu tham khảo
Ahadi, A., Lister, R., Vihavainen, A. (2016). On the number of attempts students made on some online programming exercises during semester and their subsequent performance on final exam questions. In: Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education, pp. 218–223
Alsawaier, R. S. (2018). The effect of gamification on motivation and engagement. The International Journal of Information and Learning Technology. https://doi.org/10.1108/IJILT-02-2017-0009.
Bai, S., Hew, K. F., & Huang, B. (2020). Does gamification improve student learning outcome? evidence from a meta-analysis and synthesis of qualitative data in educational contexts. Educational Research Review, 30, 100322.
Branch, R. M. (2009). Instructional design: The ADDIE approach (Vol. 722). Berlin: Springer.
Briffa, M., Jaftha, N., Loreto, G., Pinto, F. C. M., Chircop, T., & Hill, C. (2020). Improved students’ performance within gamified learning environment: A meta-analysis study. International Journal of Education and Research, 8(1), 223–244.
Cairns, P. (2019). Doing better statistics in human-computer interaction. Cambridge: Cambridge University Press.
Clark, R. E. (1983). Reconsidering research on learning from media. Review of educational research, 53(4), 445–459.
Creswell, J. W., & Creswell, J. D. (2017). Research design: qualitative, quantitative, and mixed methods approaches. New York: Sage publications.
Deterding, S., Dixon, D., Khaled, R., Nacke, L. (2011). From game design elements to gamefulness: defining gamification. In: Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments, pp. 9–15 . ACM
Dichev, C., & Dicheva, D. (2017). Gamifying education: What is known, what is believed and what remains uncertain: a critical review. International Journal of Educational Technology in Higher Education, 14(1), 9.
Dick, W., Carey, L., Carey, J. O. (2005). The systematic design of instruction.
Dragicevic, P. (2016). Fair statistical communication in hci. In: Modern Statistical Methods for HCI, pp. 291–330. Springer
Estey, A., Coady, Y. (2016). Can interaction patterns with supplemental study tools predict outcomes in CS1? Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education—ITiCSE ’16, 236–241
Fonseca, S. C., Pereira, F. D., Oliveira, E. H., Oliveira, D. B., Carvalho, L. S., Cristea, A. I. (2020). Automatic subject-based contextualisation of programming assignment lists. In Proceedings of The 13th International Conference on Educational Data Mining (EDM 2020).
Galvão, L., Fernandes, D., Gadelha, B.(2016). Juiz online como ferramenta de apoio a uma metodologia de ensino híbrido em programação. In: Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), vol.27, p. 140
Goldstein, E. B. (2014). Cognitive Psychology: Connecting Mind. Nelson Education: Research and Everyday Experience.
Grangeia, T.d.A.G., de Jorge, B., Cecílio-Fernandes, D., Tio, R.A., de Carvalho-Filho, M.A. (2019). Learn+ fun! social media and gamification sum up to foster a community of practice during an emergency medicine rotation. Health Professions Education, 5(4), 321–335.
Hallifax, S., Serna, A., Marty, J.-C., & Lavoué, É. (2019). Adaptive gamification in education: A literature review of current trends and developments. In M. Scheffel, J. Broisin, V. Pammer-Schindler, A. Ioannou, & J. Schneider (Eds.), Transforming learning with meaningful technologies (pp. 294–307). Cham: Springer.
Hamari, J., Koivisto, J., Sarsa, H. (2014). Does gamification work?–a literature review of empirical studies on gamification. In: 2014 47th Hawaii International Conference on System Sciences, pp. 3025–3034 . Ieee
Hanus, M. D., & Fox, J. (2015). Assessing the effects of gamification in the classroom: A longitudinal study on intrinsic motivation, social comparison, satisfaction, effort, and academic performance. Computers& Education, 80, 152–161. https://doi.org/10.1016/j.compedu.2014.08.019.
Helmefalk, M. (2019). An interdisciplinary perspective on gamification: Mechanics, psychological mediators and outcomes. International Journal of Serious Games, 6(1), 3–26.
Hernández, M. I. O., Lezama, R. M., Gómez, S. M.(2021). Work-in-progress: The road to learning, using gamification. In: 2021 IEEE Global Engineering Education Conference (EDUCON), pp. 1393–1397 . IEEE
Huang, R., Ritzhaupt, A. D., Sommer, M., Zhu, J., Stephen, A., Valle, N., et al. (2020). The impact of gamification in educational settings on student learning outcomes: A meta-analysis. Educational Technology Research and Development. https://doi.org/10.1007/s11423-020-09807-z.
Hyrynsalmi, S., Smed, J., Kimppa, K. (2017). The dark side of gamification: How we should stop worrying and study also the negative impacts of bringing game design elements to everywhere. In: GamiFIN, pp. 96–104
Jafari, M., & Ansari-Pour, N. (2019). Why, when and how to adjust your p values? Cell Journal (Yakhteh), 20(4), 604.
Keselman, H., Algina, J., Wilcox, R. R., & Kowa, R. K. (2000). Testing repeated measures hypotheses when covariance matrices are heterogeneous: Revisiting the robustness of the welch-james test again. Educational and Psychological Measurement, 60(6), 925–938.
Klock, A. C. T., Gasparini, I., Pimenta, M. S., & Hamari, J. (2020). Tailored gamification: A review of literature. International Journal of Human-Computer Studies. https://doi.org/10.1016/j.ijhcs.2020.102495.
Koivisto, J., & Hamari, J. (2014). Demographic differences in perceived benefits from gamification. Computers in Human Behavior, 35, 179–188. https://doi.org/10.1016/j.chb.2014.03.007.
Koivisto, J., & Hamari, J. (2019). The rise of motivational information systems: A review of gamification research. International Journal of Information Management, 45, 191–210. https://doi.org/10.1016/j.ijinfomgt.2018.10.013.
Kotrlik, J., & Williams, H. (2003). The incorporation of effect size in information technology, learning, information technology, learning, and performance research and performance research. Information Technology, Learning, and Performance Journal, 21(1), 1.
Kowalchuk, R. K., Keselman, H., & Algina, J. (2003). Repeated measures interaction test with aligned ranks. Multivariate Behavioral Research, 38(4), 433–461.
Kyewski, E., & Krämer, N. C. (2018). To gamify or not to gamify? an experimental field study of the influence of badges on motivation, activity, and performance in an online learning course. Computers& Education, 118, 25–37.
Landers, R. N., & Landers, A. K. (2014). An empirical test of the theory of gamified learning: The effect of leaderboards on time-on-task and academic performance. Simulation& Gaming, 45(6), 769–785.
Liu, D., Santhanam, R., & Webster, J. (2017). Toward meaningful engagement: A framework for design and research of gamified information systems. MIS quarterly, 41(4), 1011–1034.
Lopez, C. E., Tucker, C. S. (2021). Adaptive gamification and its impact on performance. In: International Conference on Human-Computer Interaction, pp. 327–341 . Springer
Mair, P., Wilcox, R. (2018). Robust statistical methods using wrs2. The WRS2 Package.
Mavletova, A. (2015). A gamification effect in longitudinal web surveys among children and adolescents. International Journal of Market Research, 57(3), 413–438.
Mitchell, R., Schuster, L., & Drennan, J. (2017). Understanding how gamification influences behaviour in social marketing. Australasian Marketing Journal (AMJ), 25(1), 12–19.
Mustafa, A. S., Karimi, K. (2021). Enhancing gamified online learning user experience (ux): A systematic literature review of recent trends. Human-Computer Interaction and Beyond-Part I, 74–99
Nacke, L. E., & Deterding, C. S. (2017). The maturing of gamification research. Computers in Human Behaviour. https://doi.org/10.1016/j.chb.2016.11.062.
Palomino, P.T., Toda, A.M., Oliveira, W., Cristea, A.I., Isotani, S. (2019). Narrative for gamification in education: why should you care? In: 2019 IEEE 19th International Conference on Advanced Learning Technologies (ICALT), vol. 2161, pp. 97–99 . IEEE
Palomino, P., Toda, A., Rodrigues, L., Oliveira, W., Isotani, S.(2020). From the lack of engagement to motivation: Gamification strategies to enhance users learning experiences. In: 2020 19th Brazilian Symposium on Computer Games and Digital Entertainment (SBGames)-GranDGames BR Forum, pp. 1127–1130
Pereira, F. D., Fonseca, S. C., Oliveira, E. H., Cristea, A. I., Bellhäuser, H., Rodrigues, L., Oliveira, D. B., Isotani, S., Carvalho, L. S. (2021). Explaining individual and collective programming students’ behaviour by interpreting a black-box predictive model. IEEE Access
Pereira, F. D., Oliveira, E. H., Oliveira, D. B., Cristea, A. I., Carvalho, L. S., Fonseca, S. C., Toda, A., Isotani, S. (2020). Using learning analytics in the amazonas: Understanding students’ behaviour in introductory programming. British Journal of Educational Technology. https://doi.org/10.1111/bjet.12953.
Pereira, F. D., Toda, A., Oliveira, E. H., Cristea, A. I., Isotani, S., Laranjeira, D., Almeida, A., Mendonça, J. (2020). Can we use gamification to predict students’ performance? a case study supported by an online judge. In: International Conference on Intelligent Tutoring Systems, pp. 259–269. Springer
Putz, L.-M., Hofbauer, F., & Treiblmaier, H. (2020). Can gamification help to improve education? Findings from a longitudinal study. Computers in Human Behavior. https://doi.org/10.1016/j.chb.2020.106392.
R Core Team. (2020). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. R Foundation for Statistical Computing. Retrieved from https://www.R-project.org/
Rodrigues, L., Oliveira, W., Toda, A., Palomino, P., Isotani, S. (2019). Thinking inside the box: How to tailor gamified educational systems based on learning activities types. In: Proceedings of the Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação—SBIE)
Rodrigues, L., Palomino, P.T., Toda, A.M., Klock, A.C., Oliveira, W., Avila-Santos, A.P., Gasparini, I., Isotani, S. (2021). Personalization improves gamification: Evidence from a mixed-methods study. Proceedings of the ACM on Human-Computer Interaction 5(CHI PLAY), 1–25
Rodrigues, L., Toda, A. M., Oliveira, W., Palomino, P. T., Avila-Santos, A. P., Isotani, S. (2021). Gamification works, but how and to whom? an experimental study in the context of programming lessons. In: Proceedings of the 52nd ACM Technical Symposium on Computer Science Education, pp. 184–190
Rodrigues, L., Toda, A. M., Oliveira, W., Palomino, P. T., Isotani, S. (2020). Just beat it: Exploring the influences of competition and task-related factors in gamified learning environments. In: Anais do XXXI Simpósio Brasileiro de Informática na Educação, pp. 461–470 . SBC
Rodrigues, L., Toda, A. M., Palomino, P. T., Oliveira, W., Isotani, S. (2020). Personalized gamification: A literature review of outcomes, experiments, and approaches. In: Eighth International Conference on Technological Ecosystems for Enhancing Multiculturality, pp. 699–706
Rowland, C. A. (2014). The effect of testing versus restudy on retention: A meta-analytic review of the testing effect. Psychological Bulletin, 140(6), 1432.
RStudio Team. (2019). RStudio: Integrated Development Environment for R. RStudio, Inc., Boston, MA . RStudio, Inc. Retrieved from http://www.rstudio.com/
Sailer, M., & Homner, L. (2019). The gamification of learning: A meta-analysis. Educational Psychology Review. https://doi.org/10.1007/s10648-019-09498-w.
Sanchez, D. R., Langer, M., & Kaur, R. (2020). Gamification in the classroom: Examining the impact of gamified quizzes on student learning. Computers& Education, 144, 103666. https://doi.org/10.1016/j.compedu.2019.103666.
Santana, B. L., Bittencourt, R. A. (2018). Increasing motivation of cs1 non-majors through an approach contextualized by games and media. In: 2018 IEEE Frontiers in Education Conference (FIE), pp. 1–9 . IEEE
Seaborn, K., & Fels, D. I. (2015). Gamification in theory and action: A survey. International Journal of human-computer studies, 74, 14–31.
Toda, A. M., Klock, A. C., Oliveira, W., Palomino, P. T., Rodrigues, L., Shi, L., Bittencourt, I., Gasparini, I., Isotani, S., Cristea, A. I. (2019). Analysing gamification elements in educational environments using an existing gamification taxonomy. Smart Learning Environments, 6(1), 16.
Toda, A., Pereira, F. D., Klock, A. C. T., Rodrigues, L., Palomino, P., Oliveira, W., Oliveira, E. H. T., Gasparini, I., Cristea, A. I., Isotani, S. (2020). For whom should we gamify? insights on the users intentions and context towards gamification in education. In: Anais do XXXI Simpósio Brasileiro de Informática na Educação, pp. 471–480 . SBC
Toda, A. M., Valle, P. H. D., & Isotani, S. (2018). The dark side of gamification: An overview of negative effects of gamification in education. In A. I. Cristea, I. I. Bittencourt, & F. Lima (Eds.), Higher education for all. From challenges to novel technology-enhanced solutions (pp. 143–156). Cham: Springer.
Tondello, G. F. (2019). Dynamic personalization of gameful interactive systems. PhD thesis, University of Waterloo
Tondello, G. F., Orji, R., Nacke, L. E. (2017). Recommender systems for personalized gamification. In: Adjunct Publication of the 25th Conference on User Modeling, Adaptation and Personalization, pp. 425–430 . https://doi.org/10.1145/3099023.3099114.ACM
Tsay, C.H.-H., Kofinas, A. K., Trivedi, S. K., & Yang, Y. (2020). Overcoming the novelty effect in online gamified learning systems: An empirical evaluation of student engagement and performance. Journal of Computer Assisted Learning, 36(2), 128–146.
Van Roy, R., & Zaman, B. (2018). Need-supporting gamification in education: An assessment of motivational effects over time. Computers& Education, 127, 283–297.
von Wangenheim, C. G., & von Wangenheim, A. (2012). Ensinando computação com jogos. Florianópolis, SC, Brasil: Bookess Editora.
Vornhagen, J. B., Tyack, A., Mekler, E. D. (2020). Statistical significance testing at chi play: Challenges and opportunities for more transparency. In: Proceedings of the Annual Symposium on Computer-Human Interaction in Play, pp. 4–18
Wilcox, R. (2017). Introduction to robust estimation and hypothesis testing (4th ed.). Amsterdam: Elsevier.
Wilcox, R. R., & Tian, T. S. (2011). Measuring effect size: A robust heteroscedastic approach for two or more groups. Journal of Applied Statistics, 38(7), 1359–1368.
Yuen, K. K. (1974). The two-sample trimmed T for unequal population variances. Biometrika, 61(1), 165–170.