Automated feedback for participants of hands-on cybersecurity training

Agudo, I., Rios, R., & Nieto, A. (2019). Personalized computer security tasks with automatic evaluation and feedback. Proceedings of the 2019 SIGED International conference on information systems education and research. Association for Information Systems (AIS). Retrieved from https://www.nics.uma.es/pub/papers/1835.pdf Arends, H., Keuning, H., Heeren, B., & Jeuring, J. (2017). An intelligent tutor to learn the evaluation of microcontroller I/O programming expressions. In Proceedings of the 17th Koli calling international conference on computing education research (pp. 2–9). New York, NY, USA: Association for computing machinery. Retrieved from https://doi.org/10.1145/3141880.3141884 Baniassad, E., Zamprogno, L., Hall, B., & Holmes, R. (2021). Stop the (autograder) insanity: Regression penalties to deter autograder overreliance. In Proceedings of the 52nd ACM Technical symposium on computer science education (pp. 1062–1068). New York, NY, USA: Association for computing machinery. Retrieved from. https://doi.org/10.1145/3408877.3432430 Bezáková, I., Hemaspaandra, E., Lieberman, A., Miller, H., & Narváez, D. E. (2020). Prototype of an automated feedback tool for intro CS theory. In Proceedings of the 51st ACM Technical symposium on computer science education (p. 1311). New York, NY, USA: Association for computing machinery. Retrieved from https://doi.org/10.1145/3328778.3372598 Bodily, R., & Verbert, K. (2017). Review of research on student-facing learning analytics dashboards and educational recommender systems. IEEE Transactions on Learning Technologies, 10(4), 405–418. Retrieved from https://doi.org/10.1109/TLT.2017.2740172 Bruzual, D., Montoya Freire, M. L., & Di Francesco, M. (2020). Automated assessment of android exercises with cloud-native technologies. In Proceedings of the 2020 ACM Conference on innovation and technology in computer science education (pp. 40–46). New York, NY, USA: Association for computing machinery. Retrieved from https://doi.org/10.1145/3341525.3387430 Buffardi, K., & Edwards, S. H. (2015). Reconsidering automated feedback: A test-driven approach. In Proceedings of the 46th ACM Technical symposium on computer science education (pp. 416–420). New York, NY, USA: Association for computing machinery. Retrieved from https://doi.org/10.1145/2676723.2677313 Carifio, J., & Perla, R. (2008). Resolving the 50-year debate around using and misusing Likert scales (vol. 42 No. 12). Retrieved from https://doi.org/10.1111/j.1365-2923.2008.03172.x Cavalcanti, A. P., Barbosa, A., Carvalho, R., Freitas, F., Tsai, Y. -S., Gaševìc, D., & Mello, R. F. (2021). Automatic feedback in online learning environments: A systematic literature review. Computers and Education: Artificial Intelligence, 2, 100027. Retrieved from https://doi.org/10.1016/j.caeai.2021.100027 Chen, H., Ciborowska, A., & Damevski, K. (2019). Using automated prompts for student reflection on computer security concepts. In Proceedings of the 2019 ACM Conference on innovation and technology in computer science education (pp. 506–512). New York, NY, USA: Association for computing machinery. Retrieved from https://doi.org/10.1145/3304221.3319731 Chothia, T., Holdcroft, S., Radu, A. -I., & Thomas, R. J. (2017). Jail, hero or drug lord? Turning a cyber security course into an 11 week choose your own adventure story. In 2017 USENIX Workshop on Advances in Security Education (ASE 17) (pp. 1–11). Berkeley, CA, USA: USENIX association. Retrieved from https://www.usenix.org/conference/ase17/workshop-program/presentation/chothia Chow, S., Yacef, K., Koprinska, I., & Curran, J. (2017). Automated data-driven hints for computer programming students. In Adjunct publication of the 25th Conference on user modeling, adaptation and personalization (pp. 5–10). New York, NY, USA: Association for computing machinery. Retrieved from https://doi.org/10.1145/3099023.3099065 Chung, K. (2017). Live lesson: Lowering the barriers to capture the flag administration and participation. In 2017 USENIX Workshop on Advances in Security Education (ASE 17). Vancouver, BC: USENIX Association. Retrieved from https://www.usenix.org/conference/ase17/workshop-program/presentation/chung Codio (2019). Report: 2019 Survey of CS teaching. Online, Accessed 3, Oct 2023. Retrieved from https://www.codio.com/research/2019-computer-science-teaching-survey Cordova, L., Carver, J., Gershmel, N., & Walia, G. (2021). A comparison of inquiry-based conceptual feedback vs. traditional detailed feedback mechanisms in software testing education: An empirical investigation. In Proceedings of the 52nd ACM Technical symposium on computer science education (pp. 87–93). New York, NY, USA: Association for computing machinery. Retrieved from https://doi.org/10.1145/3408877.3432417 DelftStack (2023). Get command history in MySQL. Online, Accessed 3 Oct 2023. Retrieved from https://www.delftstack.com/howto/mysql/mysql-command-history/ Demčák, J. (2021). Web application for automated feedback to participants of cybersecurity training (Master thesis, Masaryk University, Faculty of Informatics). Retrieved from https://is.muni.cz/th/a4pjx/?lang=en Denny, P., Becker, B. A., Craig, M., Wilson, G., & Banaszkiewicz, P. (2019). Research this! questions that computing educators most want computing education researchers to answer. In Proceedings of the 2019 ACM Conference on international computing education research (pp. 259–267). New York, NY, USA: Association for computing machinery. Retrieved from https://doi.org/10.1145/3291279.3339402 Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015). Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1), 114. Retrieved from https://doi.org/10.1037/xge0000033 Doupé, A., Egele, M., Caillat, B., Stringhini, G., Yakin, G., Zand, A., & . . . Vigna, G. (2011). Hit’Em Where It Hurts: A live security exercise on cyber situational awareness. In Proceedings of the 27th Annual computer security applications conference (pp. 51–61). New York, NY, USA: ACM. Retrieved from https://doi.org/10.1145/2076732.2076740 Edwards, S. H., & Pérez Quiñones, M. A. (2008). Web-CAT: Automatically grading programming assignments. SIGCSE Bull., 40(3), 328. Retrieved from https://doi.org/10.1145/1597849.1384371 Elasticsearch (2023). What is the ELK Stack? Online, Accessed 3 Oct 2023. Retrieved from https://www.elastic.co/what-is/elk-stack Elkherj, M., & Freund, Y. (2014). A system for sending the right hint at the right time. In Proceedings of the First ACM Conference on Learning Scale (pp. 219–220). New York, NY, USA: Association for computing machinery. Retrieved from https://doi.org/10.1145/2556325.2567864 Fraser, C. A., Ngoon, T. J., Weingarten, A. S., Dontcheva, M., & Klemmer, S. (2017). CritiqueKit: A mixed-initiative, real-time interface for improving feedback. In Adjunct publication of the 30th Annual ACM Symposium on user interface software and technology (pp. 7–9). Association for Computing Machinery. Retrieved from https://doi.org/10.1145/3131785.3131791 Ghafarian, A. (2019). Using kali linux security tools to create laboratory projects for cybersecurity education. In Proceedings of the Future Technologies Conference (FTC) 2018 (pp. 358–367). Cham: Springer international publishing. Retrieved from https://doi.org/10.1007/978-3-030-02683-7_25 Guzdial, M., & du Boulay, B. (2019). The history of computing education research. In S. A. Fincher & A. V. Robins (Eds.), The cambridge handbook of computing education research (pp. 11–39). Cambridge, United Kingdom: Cambridge University Press. Retrieved from https://doi.org/10.1017/9781108654555 Haldeman, G., Tjang, A., Babeş-Vroman, M., Bartos, S., Shah, J., Yucht, D., & Nguyen, T. D. (2018). Providing meaningful feedback for autograding of programming assignments. In Proceedings of the 49th ACM Technical symposium on computer science education (pp. 278–283). New York, NY, USA: Association for computing machinery. Retrieved from https://doi.org/10.1145/3159450.3159502 Hall, J. G., Mohanty, A., Murarisetty, P., Nguyen, N. D., Bahamòn, J. C., Ramaprasad, H., & Sridhar, M. (2022). Criminal investigations: An interactive experience to improve student engagement and achievement in cybersecurity courses. In Proceedings of the 53rd ACM Technical symposium on computer science education (vol. 1 pp. 696–702). New York, NY, USA: Association for computing machinery. Retrieved from https://doi.org/10.1145/3478431.3499417 Hao, J., Shu, Z., & von Davier, A. (2015). Analyzing process data from game/scenario-based tasks: An edit distance approach. Journal of Educational Data Mining, 7(1), 33–50. Retrieved from https://jedm.educationaldatamining.org/index.php/JEDM/article/view/JEDM072 Harden, J., Gusukuma, L., Bart, A. C., & Kafura, D. (2021). A specification language for matching mistake patterns with feedback. In Proceedings of the 52nd ACM Technical symposium on computer science education (pp. 555–561). New York, NY, USA: Association for computing machinery. Retrieved from https://doi.org/10.1145/3408877.3432481 Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81-112. Retrieved from https://doi.org/10.3102/003465430298487 Heckman, J. J. (2010). Selection bias and self-selection. In S.N. Durlauf & L. E. Blume (Eds.), Microeconometrics (pp. 242–266). London: Palgrave Macmillan UK. Retrieved from https://doi.org/10.1057/9780230280816_29 Joint task force on cybersecurity education (2017). Cybersecurity curricular guideline. Online, Accessed 3 Oct 2023. Retrieved from https://cybered.acm.org Keuning, H., Jeuring, J., & Heeren, B. (2018). A systematic literature review of automated feedback generation for programming exercises. ACM Transactions on Computing Education19(1), 3:1–3:43. Retrieved from https://doi.org/10.1145/3231711 Kölling, M., & McKay, F. (2016). Heuristic evaluation for novice programming systems. ACM Transactions on Computing Education, 16(3), 1–30. Retrieved from https://doi.org/10.1145/2872521 Kont, M., Pihelgas, M., Maennel, K., Blumbergs, B., & Lepik, T. (2017). Frankenstack: Toward real-time red team feedback. In 2017 IEEE Military Communications Conference, MILCOM 2017 (pp. 400–405). New York, NY, USA: IEEE. Retrieved from https://doi.org/10.1109/MILCOM.2017.8170852 Kovanović, V., Joksimović, S., Gašević, D., Hatala, M., & Siemens, G. (2017). Content analytics: The definition, scope, and an overview of published research. In C. Lang, G. Siemens, A. Wise, & D. Gašević (Eds.), Handbook of Learning Analytics (pp. 77–92). Society for Learning Analytics Research (SoLAR). Retrieved from https://doi.org/10.18608/hla17 Kulkarni, C. E., Bernstein, M. S., & Klemmer, S. R. (2015). PeerStudio: Rapid peer feedback emphasizes revision and improves performance. In Proceedings of the 2nd (2015) ACM Conference on learning scale (pp. 75–84). Retrieved from https://doi.org/10.1145/2724660.2724670 Lepri, B., Oliver, N., Letouzé, E., Pentland, A., & Vinck, P. (2018). Fair, transparent, and accountable algorithmic decision-making processes. Philosophy & Technology, 31(4), 611–627. Retrieved from https://doi.org/10.1007/s13347-017-0279-x Lyon, G. (n.d.). Nmap reference guide. Online, Accessed 3 Oct 2023. Retrieved from https://nmap.org/book/man-briefoptions.html Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an “early warning system” for educators: A proof of concept. Computers & Education, 54(2), 588–599. Retrieved from https://doi.org/10.1016/j.compedu.2009.09.008 Malik, A., Wu, M., Vasavada, V., Song, J., Coots, M., Mitchell, J., &. . . Piech, C. (2021). Generative grading: near human-level accuracy for automated feedback on richly structured problems. In Proceedings of the 14th International conference on educational data mining. International educational data mining society. Retrieved from https://educationaldatamining.org/EDM2021/virtual/static/pdf/EDM21_paper_94.pdf Malmi, L., Utting, I., Ko, A. J. (2019). Tools and environments. In S. A. Fincher & A. V. Robins (Eds.), The cambridge handbook of computing education research (pp. 639–662). Cambridge, United Kingdom: Cambridge University Press. Retrieved from https://doi.org/10.1017/9781108654555.022 Mandiant (2013). APT1: Exposing one of china’s cyber espionage units (Tech. Rep.). FireEye, Inc. Retrieved from https://www.mandiant.com/resources/apt1-exposing-one-of-chinas-cyber-espionage-units Marchiori, A. (2022). Labtool: A command-line interface lab assistant and assessment tool. In Proceedings of the 53rd ACM Technical symposium on computer science education (vol. 1 pp. 1–7). New York, NY, USA: Association for computing machinery. Retrieved from https://doi.org/10.1145/3478431.3499285 Marwan, S., Gao, G., Fisk, S., Price, T. W., & Barnes, T. (2020). Adaptive immediate feedback can improve novice programming engagement and intention to persist in computer science. In Proceedings of the 2020 ACM Conference on international computing education research (pp. 194–203). New York, NY, USA: Association for computing machinery. Retrieved from https://doi.org/10.1145/3372782.3406264 Masaryk University (2021). The listing of all cybersecurity games and common instructions. Online, Accessed 3 Oct 2023. Retrieved from https://gitlab.ics.muni.cz/muni-kypo-trainings/games/all-games-index Masaryk University (2023a). KYPO Cyber range platform. Online, Accessed 3 Oct 2023. Retrieved from https://docs.crp.kypo.muni.cz Masaryk University (2023b). KYPO Training Feedback. Online, Accessed 3 Oct 2023. Retrieved from https://gitlab.ics.muni.cz/muni-kypo-crp/backend-java/kypo-training-feedback Masaryk University (2023c). Training command graph visualization. Online, Accessed 3 Oct 2023. Retrieved from https://gitlab.ics.muni.cz/muni-kypo-crp/frontend-angular/components/kypo-command-visualizations Masaryk University (2023d). KYPO Cyber range platform – visualizations for linear training. Online, Accessed 3 Oct 2023. Retrieved from https://docs.crp.kypo.muni.cz/user-guide-basic/training-agenda/visualizations/visualizations-for-linear Miller, J., & Ulrich, R. (2019). The quest for an optimal alpha. PLOS ONE, 14 . Retrieved from https://doi.org/10.1371/journal.pone.0208631 Mouratidis, A., Lens, W., & Vansteenkiste, M. (2010). How you provide corrective feedback makes a difference: The motivating role of communicating in an autonomy-supporting way. Journal of Sport and Exercise Psychology, 32(5), 619–637. Retrieved from https://doi.org/10.1123/jsep.32.5.619 Nicol, D. J., & Macfarlane-Dick, D. (2006). Formative assessment and selfregulated learning: a model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199–218. Retrieved from https://doi.org/10.1080/03075070600572090 Norman, G. (2010). Likert scales, levels of measurement and the “laws” of statistics. Advances in health sciences education, 15(5), 625–632. Retrieved from https://doi.org/10.1007/s10459-010-9222-y OConnor, T. (2022). HELO darkSide: Breaking free from katas and embracing the adversarial mindset in cybersecurity education. In Proceedings of the 53rd ACM Technical symposium on computer science education (vol. 1 p. 710–716). New York, NY, USA: Association for computing machinery. Retrieved from https://doi.org/10.1145/3478431.3499404 Offensive Security (2023). Kali linux. Online, Accessed 3 Oct 2023. Retrieved from https://www.kali.org Offensive Security (OffSec Services Limited) (2023). Metasploit Unleashed. Online, Accessed 3 Oct 2023. Retrieved from https://www.offensive-security.com/metasploit-unleashed Openwall (n.d.). John the Ripper password cracker. Online, Accessed 3 Oct 2023. Retrieved from https://www.openwall.com/john Ošlejšek, R., Rusňák, V., Burská, K., Švábenský, V., & Vykopal, J. (2019). Visual feedback for players of multi-level capture the flag games: Field usability study. In Proceedings of the 16th IEEE Symposium on visualization for cyber security (pp. 1–11). New York, NY, USA: IEEE. Retrieved from https://doi.org/10.1109/VizSec48167.2019.9161386 Ošlejšek, R., Rusňák, V., Burská, K., Švábenský, V., Vykopal, J., & Čegan, J. (2020). Conceptual model of visual analytics for handson cybersecurity training. IEEE Transactions on Visualization and Computer Graphics, 27(8). Retrieved from https://doi.org/10.1109/TVCG.2020.2977336 Pardo, A., Jovanovic, J., Dawson, S., Gašević, D., Mirriahi, N. (2019). Using learning analytics to scale the provision of personalised feedback. British Journal of Educational Technology, 50(1), 128–138. Retrieved from https://doi.org/10.1111/bjet.12592 Peña-Ayala, A. (2014). Educational data mining: A survey and a data miningbased analysis of recent works. Expert Systems with Applications, 41(4, Part 1), 1432–1462. Retrieved from https://doi.org/10.1016/j.eswa.2013.08.042 Petty, G. (2009). Teaching today: A practical guide. Nelson Thornes. Retrieved from https://geoffpetty.com/geoffs-books/teaching-today Pihelgas, M., & Kont, M. (2021). Frankenstack: Real-time cyberattack detection and feedback system for technical cyber exercises. In 2021 IEEE International Conference on Cyber Security and Resilience (CSR) (pp. 396–402). Retrieved from https://doi.org/10.1109/CSR51186.2021.9527923 Pokhrel, S., & Chhetri, R. (2021). A literature review on impact of COVID-19 Pandemic on teaching and Learning. Higher Education for the Future, 8(1), 133–141. Retrieved from https://doi.org/10.1177/2347631120983481 Price, T., Zhi, R., Barnes, T. (2017). Evaluation of a data-driven feedback algorithm for open-ended programming. International conference on educational data mining. Retrieved from https://educationaldatamining.org/EDM2017/proc_files/papers/paper_36.pdf Price, T. W., Dong, Y., & Lipovac, D. (2017). ISnap: Towards intelligent tutoring in novice programming environments. Proceedings of the 2017 ACM SIGCSE Technical symposium on computer science education (pp. 483–488). New York, NY, USA: Association for computing machinery. Retrieved from https://doi.org/10.1145/3017680.3017762 Price, T. W., Dong, Y., Zhi, R., PaaSSen, B., Lytle, N., Cateté, V., & Barnes, T. (2019). A comparison of the quality of data-driven programming hint generation algorithms. International Journal of Artificial Intelligence in Education, 29(3), 368–395. Retrieved from https://doi.org/10.1007/s40593-019-00177-z R Core Team (2023). The R Project for statistical computing. Vienna, Austria. Retrieved from https://www.r-project.org/ Rowe, E., Almeda, M. V., Asbell-Clarke, J., Scruggs, R., Baker, R., Bardar, E., & Gasca, S. (2021). Assessing implicit computational thinking in Zoombinis puzzle gameplay. Computers in Human Behavior, 120, 106707. Retrieved from https://doi.org/10.1016/j.chb.2021.106707 Saldaña, J. (2021). The coding manual for qualitative researchers. Sage publications. Retrieved from https://books.google.com/books?id=RwcVEAAAQBAJ Schmider, E., Ziegler, M., Danay, E., Beyer, L., & Buhner, M. (2010). Is it really robust? Reinvestigating the robustness of ANOVA against the normal distribution assumption. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 6, 147–151. Retrieved from https://doi.org/10.1027/1614-2241/a000016 Schwendimann, B. A., Rodrìguez-Triana, M. J., Vozniuk, A., Prieto, L. P., Boroujeni, M. S., Holzer, A., &. . . Dillenbourg, P. (2017). Perceiving learning at a glance: A systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies, 10(1), 30–41. Retrieved from https://doi.org/10.1109/TLT.2016.2599522 Sehl, S., & Vaniea, K. (2018). Permission Impossible: Teaching Firewall Configuration in a Game Environment. In European Workshop on Usable Security (EuroUSEC). Retrieved from https://doi.org/10.14722/eurousec.2018.23006 Shephard, K. (2019). Higher education pedagogy. In S. A. Fincher & A. V. Robins (Eds.), The Cambridge Handbook of Computing Education Research (pp. 276–291). Cambridge, United Kingdom: Cambridge University Press. Retrieved from https://doi.org/10.1017/9781108654555.011 Shute, V. J. (2008). Focus on Formative Feedback. Review of Educational Research, 78(1), 153–189. Retrieved from https://doi.org/10.3102/0034654307313795 Švábenský, V., Vykopal, J., & Čeleda, P., Dovjak, J. (2023). Supplementary Materials: Automated Feedback for Participants of Hands-on Cybersecurity Training. Zenodo: Retrieved from. https://doi.org/10.5281/zenodo.8405210 Švábenský, V., Vykopal, J., Čeleda, P. (2020). What are cybersecurity education papers about? a systematic literature review of SIGCSE and ITiCSE conferences. In Proceedings of the 51st ACM Technical symposium on computer science education (pp. 2–8). New York, NY, USA: Association for computing machinery. Retrieved from https://doi.org/10.1145/3328778.3366816 Švábenský, V., Vykopal, J., Čeleda, P., & Kraus, L. (2022). Applications of educational data mining and learning analytics on data from cybersecurity training. Springer Education and Information Technologies: Retrieved from. https://doi.org/10.1007/s10639-022-11093-6 Švábenský, V., Vykopal, J., Cermak, M., & Laštovička, M. (2018). Enhancing cybersecurity skills by creating serious games. Proceedings of the 23rd Annual ACM Conference on innovation and technology in computer science education (pp. 194–199). New York, NY, USA: Association for computing machinery. Retrieved from https://doi.org/10.1145/3197091.3197123 Švábenský, V., Vykopal, J., Seda, P., & Čeleda, P. (2021). Dataset of shell commands used by participants of hands-on cybersecurity training. Elsevier Data in Brief, 38 . Retrieved from https://doi.org/10.1016/j.dib.2021.107398 Švábenský, V., Vykopal, J., Tovarňák, D., & Čeleda, P. (2021). Toolset for collecting shell commands and its application in hands-on cybersecurity training. In Proceedings of the 51st IEEE Frontiers in education conference (pp. 1–9). New York, NY, USA: IEEE. Retrieved from https://doi.org/10.1109/FIE49875.2021.9637052 Švábenský, V., Weiss, R., Cook, J., Vykopal, J., Čeleda, P., Mache, J., & . . . Chattopadhyay, A. (2022). Evaluating two approaches to assessing student progress in cybersecurity exercises. In Proceedings of the 53rd ACM Technical symposium on computer science education. New York, NY, USA: Association for computing machinery. Retrieved from https://doi.org/10.1145/3478431.3499414 Thomas, J. W. (1993). Promoting independent learning in the middle grades: The role of instructional support practices. The Elementary School Journal, 93(5), 575-591. Retrieved from https://doi.org/10.1086/461741 Vykopal, J., Čeleda, P., Seda, P., Švábenský, V., & Tovarňák, D. (2021). Scalable learning environments for teaching cybersecurity hands-on. In Proceedings of the 51st IEEE Frontiers in education conference (pp. 1–9). New York, NY, USA: IEEE. Retrieved from https://doi.org/10.1109/FIE49875.2021.9637180 Vykopal, J., Švábenský, V., & Chang, E.-C. (2020). Benefits and Pitfalls of Using Capture the Flag Games in University Courses. In Proceedings of the 51st ACM Technical symposium on computer science education (pp. 752–758). New York, NY, USA: Association for Computing Machinery. Retrieved from https://doi.org/10.1145/3328778.3366893 Wang, W., Rao, Y., Zhi, R., Marwan, S., Gao, G., & Price, T. W. (2020). Step tutor: Supporting students through step-by-step example-based feedback. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education (pp. 391–397). New York, NY, USA: Association for computing machinery. Retrieved from https://doi.org/10.1145/3341525.3387411 Wu, H., & Leung, S.-O. (2017). Can likert scales be treated as interval scales?– A simulation study. Journal of Social Service Research, 43(4), 527–532. Retrieved from https://doi.org/10.1080/01488376.2017.1329775 Yamin, M. M., Katt, B., & Gkioulos, V. (2020). Cyber ranges and security testbeds: Scenarios, functions, tools and architecture. Computers & Security, 88, 101636. Retrieved from https://doi.org/10.1016/j.cose.2019.101636