A review on process-oriented approaches for analyzing novice solutions to programming problems
Tóm tắt
High attrition and dropout rates are common in introductory programming courses. One of the reasons students drop out is loss of motivation due to the lack of feedback and proper assessment of their progress. Hence, a process-oriented approach is needed in assessing programming progress, which entails examining and measuring students’ compilation behaviors and source codes. This paper reviews the elements of a process-oriented approach including previous studies that have used this approach. Specific metrics covered are Jadud’s Error Quotient, the Watwin Score, Probabilistic Distance to Solution, Normalized Programming State Model, and the Repeated Error Density.
Từ khóa
Tài liệu tham khảo
Adu-Manusarpong, K., Arthur, J.K., Amoako, P.Y.O. (2013). Causes of failure of students in computer programming courses: The teacher learner perspective. International Journal of Computer Applications, 7(12), 27–32.
Ahadi, A., Lister, R., Haapala, H., Vihavainen, A. (2015). Exploring machine learning methods to automatically identify students in need of assistance. In Proceedings of the Eleventh Annual International Conference on International Computing Education Research. ACM, New York, (pp. 121–130).
Ahmadzadeh, M., Elliman, D., Higgins, C. (2005). An analysis of patterns of debugging among novice computer science students. SIGCSE Bull, 37(3), 84–88.
Allevato, A., & Edwards, S.H. (2010). Discovering patterns in student activity on programming assignments. In ASEE Southeastern Section Annual Conference and Meeting.
Allevato, A., Thornton, M., Edwards, S., Perez-Quinones, M. (2008). Mining data from an automated grading and testing system by adding rich reporting capabilities. Educational Data Mining.
Altadmri, A., & Brown, N.C.C. (2015). 37 million compilations: Investigating novice programming mistakes in large-scale student data. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education. ACM, New York, (pp. 522–527).
Annamaa, A., Hansalu, A., Tonisson, E. (2015). Automatic analysis of students’ solving process in programming exercises. In IFIP TC3 Working Conference “A New Culture of Learning: Computing and next Generations”.
Becker, B.A. (2016). A new metric to quantify repeated compiler errors for novice programmers. In Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education. ACM, New York, (pp. 296–301).
Becker, B.A., Goslin, K., Glanville, G. (2018). The effects of enhanced compiler error messages on a syntax error debugging test. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education (SIGCSE ’18). ACM, New York, (pp. 640–645).
Bennedsen, J., & Caspersen, M.E. (2007). Failure rates in introductory programming. SIGCSE Bull, 39(2), 32–36.
Biggers, M., Brauer, A., Yilmaz, T. (2008). Student perceptions of computer science: A retention study comparing graduating seniors with cs leavers. In Proceedings of the 39th SIGCSE Technical Symposium on Computer Science Education. ACM, New York, (pp. 402–406).
Blikstein, P. (2011). Using learning analytics to assess students’ behavior in open-ended programming tasks. In Proceedings of the 1st International Conference on Learning Analytics and Knowledge. ACM, New York, (pp. 110–116).
Blikstein, P., Worsley, M., Piech, C., Sahami, M., Cooper, S., Koller, D. (2014). Programming pluralism: Using learning analytics to detect patterns in the learning of computer programming. Journal of the Learning Sciences, 23(4), 561–599.
Carbone, A., Hurst, J., Mitchell, I., Gunstone, D. (2009). An exploration of internal factors influencing student learning of programming. In Proceedings of the Eleventh Australasian Conference on Computing Education - Volume 95. Australian Computer Society, Inc, Darlinghurst, (pp. 25–34).
Cardell-Oliver, R (2011). How can software metrics help novice programmers? In Proceedings of the Thirteenth Australasian Computing Education Conference - Volume 114. Australian Computer Society, Inc, Darlinghurst, (pp. 55–62).
Carter, A.S., & Hundhausen, C.D (2017). Using programming process data to detect differences in students’ patterns of programming. In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE ’17). ACM, New York, (pp. 105–110).
Carter, A.S., Hundhausen, C.D., Adesope, O. (2015). The normalized programming state model: Predicting student performance in computing courses based on programming behavior. In Proceedings of the Eleventh Annual International Conference on International Computing Education Research. ACM, New York, (pp. 141–150).
Carter, A.S., Hundhausen, C.D., Adesope, O. (2017). Blending measures of programming and social behavior into predictive models of student achievement in early computing courses. ACM Transactions on Computing Education (TOCE), 17(3), 1–20.
Denny, P., Luxton-Reilly, A., Carpenter, D. (2014). Enhancing syntax error messages appears ineffectual. In Proceedings of the 2014 Conference on Innovation and Technology in Computer Science Education (ITiCSE ’14). ACM, New York, (pp. 273–278).
Edwards, S.H., & Perez-Quinones, M.A. (2008). Web-cat: Automatically grading programming assignments. SIGCSE Bull, 40(3), 328–328.
Falkner, N.J.G., & Falkner, K.E. (2012). A fast measure for identifying at-risk students in computer science. In Proceedings of the Ninth Annual International Conference on International Computing Education Research. ACM, New York, (pp. 55–62).
Fenwick, Jr., J.B., Norris, C., Barry, F.E., Rountree, J., Spicer, C.J., Cheek, S.D. (2009). Another look at the behaviors of novice programmers. SIGCSE Bull., 41(1), 296–300.
Gatchalian, E. (2014). A comparative analysis of intention-based score trajectories and update characteristics trajectories among novice programming students.
Gomes, A., & Mendes, A.J. (2007). Learning to program-difficulties and solutions. In International Conference on Engineering Education.
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112.
Heinonen, K., Hirvikoski, K., Luukkainen, M., Vihavainen, A. (2014). Using codebrowser to seek differences between novice programmers. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education. ACM, New York, (pp. 229–234).
Helme, S., & Clarke, D. (2001). Identifying cognitive engagement in the mathematics classroom. Mathematics Education Research Journal, 13(2), 133–153.
Helminen, J, Ihantola, P, Karavirta, V (2013). Recording and analyzing in-browser programming sessions. In Proceedings of the 13th Koli Calling International Conference on Computing Education Research. ACM, New York, (pp. 13–22).
Helminen, J., Ihantola, P., Karavirta, V., Malmi, L. (2012). How do students solve parsons programming problems?: An analysis of interaction traces In Proceedings of the Ninth Annual International Conference on International Computing Education Research. ACM, New York, (pp. 119–126).
Horton, D., & Craig, M. (2015). Drop, fail, pass, continue: Persistence in cs1 and beyond in traditional and inverted delivery. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education, SIGCSE ’15. ACM, New York, (pp. 235–240).
Hosseini, R., Vihavainen, A., Brusilovsky, P. (2014). Exploring problem solving paths in a java programming course. In Psychology of Programming Interest Group Conference, (pp. 65–76).
Ihantola, P., Sorva, J., Vihavainen, A. (2014). Automatically detectable indicators of programming assignment difficulty. In Proceedings of the 15th Annual Conference on Information Technology Education. ACM, New York, (pp. 33–38).
Ihantola, P., Vihavainen, A., Ahadi, A., Butler, M., Börstler, J., Edwards, S.H., Isohanni, E., Korhonen, A., Petersen, A., Rivers, K., Rubio, M.A., Sheard, J., Skupas, B., Spacco, J., Szabo, C., Toll, D. (2015). Educational data mining and learning analytics in programming: Literature review and case studies. ACM, New York, (pp. 41–63).
Jadud, M. (2006a). An exploration of novice compilation behaviour in BlueJ. PhD thesis: University of Kent.
Jadud, M.C. (2005). A first look at novice compilation behaviour using bluej. Computer Science Education, 15(1), 25–40.
Jadud, M.C. (2006b). Methods and tools for exploring novice compilation behaviour. In Proceedings of the Second International Workshop on Computing Education Research. ACM, New York, (pp. 73–84).
Johnson, M., Eagle, M., Stamper, J., Barnes, T. (2013). An algorithm for reducing the complexity of interaction networks. Educational Data Mining.
Johnson, P.M., Kou, H., Agustin, J., Chan, C., Moore, C., Miglani, J., Zhen, S., Doane, W.E.J. (2003). Beyond the personal software process: Metrics collection and analysis for the differently disciplined. In Proceedings of the 25th International Conference on Software Engineering. IEEE Computer Society, Washington, DC, (pp. 641–646).
Johnson, W.L.S.E. (1984). Intention-based diagnosis of programming errors. In Proceedings of the Fourth AAAI Conference on Artificial Intelligence. AAAI Press, (pp. 162–168).
Kasurinen, J., & Nikula, U. (2009). Estimating programming knowledge with bayesian knowledge tracing. In Proceedings of the 14th Annual ACM SIGCSE Conference on Innovation and Technology in Computer Science Education. ACM, New York, (pp. 313–317).
Kiesmueller, U., Sossalla, S., Brinda, T., Riedhammer, K. (2010). Online identification of learner problem solving strategies using pattern recognition methods. In Proceedings of the Fifteenth Annual Conference on Innovation and Technology in Computer Science Education. ACM, New York, (pp. 274–278).
Kinnunen, P., & Malmi, L. (2006). Why students drop out cs1 course? In Proceedings of the Second International Workshop on Computing Education Research. ACM, New York, (pp. 97–108).
Koprinska, I., Stretton, J., Yacef, K. (2015). Students at risk: Detection and remediation. Educational Data Mining, 512–515.
Kori, K., Pedaste, M., Tõnisson, E., Palts, T., Altin, H., Rantsus, R., Sell, R., Murtazin, K., Rüütmann, T. (2015). First-year dropout in ICT studies. In 2015 IEEE Global Engineering Education Conference (EDUCON). IEEE.
Lane, H.C., & VanLehn, K. (2005). Intention-based scoring: An approach to measuring success at solving the composition problem. In Proceedings of the 36th SIGCSE Technical Symposium on Computer Science Education. ACM, New York, (pp. 373–377).
Leinonen, J., Longi, K., Klami, A., Vihavainen, A. (2016). Automatic inference of programming performance and experience from typing patterns. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education. ACM, New York, (pp. 132–137).
Marion, B., Impagliazzo, J., St. Clair, C., Soroka, B., Whitfield, D. (2007). Assessing computer science programs: What have we learned, (pp. 131–132).
Matsuzawa, Y., Okada, K., Sakai, S. (2013). Programming process visualizer: A proposal of the tool for students to observe their programming process. In Proceedings of the 18th ACM Conference on Innovation and Technology in Computer Science Education. ACM, New York, (pp. 46–51).
McCracken, M., Almstrum, V., Diaz, D., Guzdial, M., Hagan, D., Kolikant, Y.B.-D., Laxer, C., Thomas, L., Utting, I., Wilusz, T. (2001). A multi-national, multi-institutional study of assessment of programming skills of first-year cs students. In Working Group Reports from ITiCSE on Innovation and Technology in Computer Science Education. ACM, New York, (pp. 125–180).
Mhashi, M, & Alakeel, A.L.I.M (2013). Difficulties facing students in learning computer programming skills at Tabuk University. In Proceedings of the 12th International Conference on Education and Educational Technology, Iwate.
Moström, J.E. (2011). A study of student problems in learning to program. PhD thesis: Umeå University, Department of Computing Science.
Norris, C, Barry, F, Fenwick Jr., J.B, Reid, K, Rountree, J. (2008). Clockit: Collecting quantitative data on how beginning software developers really work, (pp. 37–41). New York: ACM.
Perkins, D.N., Hancock, C., Hobbs, R., Martin, F., Simmons, R. (1986). Conditions of learning in novice programmers. Journal of Educational Computing Research, 2(1), 37–55.
Petersen, A., Spacco, J., Vihavainen, A. (2015). An exploration of error quotient in multiple contexts. In Proceedings of the 15th Koli Calling Conference on Computing Education Research. ACM, New York, (pp. 77–86).
Pettit, R., Clements, R., Mengel, S. (2012). A process for collecting and analyzing chronological data for cs1 programming assignments. In Proceedings of the International Conference on Frontiers in Education: Computer Science and Computer Engineering.
Pettit, R., Homer, J., Gee, R., Mengel, S., Starbuck, A. (2015). An empirical study of iterative improvement in programming assignments. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education. ACM, New York, (pp. 410–415).
Piech, C., Sahami, M., Koller, D., Cooper, S., Blikstein, P. (2012). Modeling how students learn to program. In Proceedings of the 43rd ACM Technical Symposium on Computer Science Education. ACM, New York, (pp. 153–160).
Rodrigo, M.M.T., Baker, R.S., Jadud, M.C., Amarra, A.C.M., Dy, T., Espejo-Lahoz, M.B.V., Lim, S.A.L., Pascua, S.A.M.S., Sugay, J.O., Tabanao, E.S. (2009). Affective and behavioral predictors of novice programmer achievement. In Proceedings of the 14th Annual ACM SIGCSE Conference on Innovation and Technology in Computer Science Education. ACM, New York, (pp. 156–160).
Rodrigo, M.M.T., & Baker, R.S.J.d. (2009). Coarse-grained detection of student frustration in an introductory programming course. In Proceedings of the Fifth International Workshop on Computing Education Research Workshop. ACM, New York, (pp. 75–80).
Rodrigo, M.M.T., Tabanao, E., Lahoz, M.B.E., Jadud, M.C. (2009). Analyzing online protocols to characterize novice java programmers. 138, 2, 177–190.
Simon, B, Lister, R, Fincher, S (2006). Multi-institutional computer science education research: A review of recent studies of novice understanding. In Proceedings. Frontiers in Education. 36th Annual Conference. IEEE.
Simon, B., Parris, J., Spacco, J. (2013). How we teach impacts student learning: Peer instruction vs. lecture in cs0. In Proceeding of the 44th ACM Technical Symposium on Computer Science Education. ACM, New York, (pp. 41–46).
Sirkiä, T., & Sorva, J. (2012). Exploring programming misconceptions: An analysis of student mistakes in visual program simulation exercises. In Proceedings of the 12th Koli Calling International Conference on Computing Education Research. ACM, New York, (pp. 19–28).
Spacco, J., Fossati, D., Stamper, J., Rivers, K. (2013). Towards improving programming habits to create better computer science course outcomes. In Proceedings of the 18th ACM Conference on Innovation and Technology in Computer Science Education. ACM, New York, (pp. 243–248).
Spacco, J., Hovemeyer, D., Pugh, W., Emad, F., Hollingsworth, J.K., Padua-Perez, N. (2006). Experiences with marmoset: Designing and using an advanced submission and testing system for programming courses. SIGCSE Bull., 38(3), 13–17.
Sudol, L.A., Rivers, K., Harris, T.K. (2012). Calculating probabilistic distance to solution in a complex problem solving domain. International Educational Data Mining Society.
Sudol-DeLyser, L.A. (2014). AbstractTutor: Increasing algorithm implementation expertise for novices through algorithmic feedback, PhD thesis: Carnegie Mellon University.
Tabanao, E.S., Rodrigo, M.M.T., Jadud, M.C. (2008). Identifying at-risk novice java programmers through the analysis of online protocols. In Philippine Computing Science Congress, (pp. 1–8).
Tabanao, E.S., Rodrigo, M.M.T., Jadud, M.C. (2011). Predicting at-risk novice java programmers through the analysis of online protocols. In Proceedings of the Seventh International Workshop on Computing Education Research. ACM, New York, (pp. 85–92).
Vee, M.H.N.C., Meyer, B., Mannock, K.L. (2006a). Understanding novice errors and error paths in object-oriented programming through log analysis. In Proceedings of workshop on educational data mining at the 8th international conference on intelligent tutoring systems, (pp. 13–20).
Vee, M.N.C., Meyer, B., Mannock, K.L. (2006b). Empirical study of novice errors and error paths in object oriented programming. In Proceedings of the 7th Annual HEA-ICS Conference.
Vihavainen, A., Helminen, J., Ihantola, P. (2014). How novices tackle their first lines of code in an ide: Analysis of programming session traces. In Proceedings of the 14th Koli Calling International Conference on Computing Education Research. ACM, New York, (pp. 109–116).
Vihavainen, A., Luukkainen, M., Ihantola, P. (2014). Analysis of source code snapshot granularity levels, (pp. 21–26). New York: ACM.
Vivian, R., Falkner, K., Falkner, N. (2013). Computer science students causal attributions for successful and unsuccessful outcomes in programming assignments. In Proceedings of the 13th Koli Calling International Conference on Computing Education Research.
Watson, C., & Li, F.W. (2014). Failure rates in introductory programming revisited. In Proceedings of the 2014 conference on Innovation and technology in computer science education.
Watson, C., Li, F.W.B., Godwin, J.L. (2012). Bluefix: Using crowd-sourced feedback to support programming students in error diagnosis and repair. In Proceedings of the 11th International Conference on Advances in Web-Based Learning (ICWL ’12). Springer-Verlag, Berlin, Heidelberg, (pp. 228–239).
Watson, C., Li, F.W.B., Godwin, J.L. (2013). Predicting performance in an introductory programming course by logging and analyzing student programming behavior. In Proceedings of the 2013 IEEE 13th International Conference on Advanced Learning Technologies. IEEE Computer Society, Washington, DC, (pp. 319–323).
Winters, T., & Payne, T. (2005). What do students know?: An outcomes-based assessment system. In Proceedings of the First International Workshop on Computing Education Research. ACM, New York, (pp. 165–172).
Worsley, M., & Blikstein, P. (2013). Programming pathways: A technique for analyzing novice programmers’ learning trajectories. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (Eds.) In Artificial Intelligence in Education. AIED 2013. Lecture Notes in Computer Science, volume 7926, Berlin, Heidelberg.