Seeing confusion through a new lens: on the impact of atoms of confusion on novices’ code comprehension

Empirical Software Engineering - Tập 28 - Trang 1-42 - 2023
José Aldo Silva da Costa1, Rohit Gheyi1, Fernando Castor2, Pablo Roberto Fernandes de Oliveira1, Márcio Ribeiro3, Baldoino Fonseca3
1Federal University of Campina Grande, Campina Grande, Brazil
2Utrecht University, Utrecht, Netherlands
3Federal University of Alagoas, Maceió, Brazil

Tóm tắt

Code comprehension is crucial for software maintenance and evolution, but it can be hindered by tiny code snippets that can confuse the developers, called atoms of confusion. Previous studies investigated how atoms impact code comprehension through the perspectives of time, accuracy, and opinions of developers. However, we need more studies evaluating other perspectives and the combination of these perspectives on a common ground through experiments. In our study, we evaluate how the eye tracking method can be used to gain new insights when we compare programs obfuscated by the atoms with functionally equivalent clarified versions. We conduct a controlled experiment with 32 novices in Python and measure their time, number of attempts, and visual effort with eye tracking through fixation duration, fixations count, and regressions count. We also conduct interviews and investigate the subjects’ difficulties with the programs. In our results, the clarified version of the code with Operator Precedence reduced the time spent in the region that contains the atom to the extent of 38.6%, and the number of answer attempts by 28%. Most subjects found the obfuscated version more difficult to solve than the clarified one, and they reported the order of precedence to be difficult to validate. By analyzing their visual effort, in the obfuscated version, we observed an increase of 47.3% in the horizontal regressions count in the atom region, making its reading more difficult. The additional atoms evaluated revealed other interesting nuances. Based on our findings, we encourage researchers to consider eye tracking combined with other perspectives to evaluate atoms of confusion and educators to favor patterns that do not impact the understanding and visual effort of undergraduates.

Tài liệu tham khảo

Basili V, Caldiera G, Rombach H (1994) The Goal Question Metric Approach. Encycl Softw Eng 2:528–532

Bednarik R, Tukiainen M (2006) An Eye-tracking Methodology for Characterizing Program Cprehension Processes. In: Proceedings of the Symposium on Eye Tracking Research & Applications, ETRA’06, pp 125–132. Association for Computing Machinery, New York

Binkley D, Davis M, Lawrie D, Maletic J, Morrell C, Sharif B (2013) The Impact of Identifier Style on Effort and Comprehension. Empir Softw Eng 18(2):219–276

Box G, Hunter JS, Hunter WG (2005) Statistics for Experimenters. Wiley-Interscience

Busjahn T, Bednarik R, Begel A, Crosby M, Paterson JH, Schulte C, Sharif B, Tamm S (2015) Eye Movements in Code Reading: Relaxing the Linear Order. In: Proceedings of the International Conference on Program Comprehension, ICPC’15, IEEE, pp 255–265

Busjahn T, Schulte C, Busjahn A (2011) Analysis of Code Reading to Gain More Insight in Program Comprehension. In: Proceedings of the Koli Calling International Conference on Computing Education Research, Koli Calling’11, pp 1–9. Association for Computing Machinery, New York

Castor F (2018) Identifying Confusing Code in Swift Programs. In: Proceedings of the CBSoft Workshop on Visualization, Evolution, and Maintenance, VEM’18. ACM, São Carlos

Cedrim D, Garcia A, Mongiovi M, Gheyi R, Sousa L, de Mello R, Fonseca B, Ribeiro M, Chávez A (2017) Understanding the impact of refactoring on smells: A longitudinal study of 23 software projects. In: Proceedings of the Joint Meeting on Foundations of Software Engineering, ESEC/FSE’17, ACM, pp 465–475

Crosby M, Scholtz J, Wiedenbeck S (2002) The Roles Beacons Play in Comprehension for Novice and Expert Programmers. In: Workshop of the Psychology of Programming Interest Group, PPIG’02, p 5. Brunel University

da Costa JAS, Gheyi R, Castor F, Roberto P, Ribeiro M, Fonseca B (2023). Seeing Confusion Through a New Lens: On the Impact of Atoms of Confusion on Novices’ Code Comprehension (Artifacts). https://doi.org/10.5281/zenodo.7650076

da Costa JAS, Gheyi R, Ribeiro M, Apel S, Alves V, Fonseca B, Medeiros F, Garcia A (2021) Evaluating Refactorings for Disciplining #ifdef Annotations: An Eye Tracking Study with Novices. Empir Softw Eng 26(5):1–35

de Oliveira B, Ribeiro M, da Costa JAS, Gheyi R, Amaral G, de Mello R, Oliveira A, Garcia A, Bonifácio R, Fonseca B (2020) Atoms of Confusion: The Eyes Do Not Lie. In: Proceedings of the Brazilian Symposium on Software Engineering, SBES’20, pp 243–252

Galley N, Betz D, Biniossek C (2015) Fixation Durations: Why are They so Highly Variable. Adv Vis Percept Res 93:83–106

Gopstein D, Fayard A-L, Apel S, Cappos J (2020) Thinking Aloud about Confusing Code: A Qualitative Investigation of Program Comprehension and Atoms of Confusion. In: Proceedings of the Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE’20, pp 605–616. Association for Computing Machinery, New York

Gopstein D, Iannacone J, Yan Y, DeLong L, Zhuang Y, Yeh MK-C, Cappos J (2017) Understanding Misunderstandings in Source Code. In: Proceedings of the Joint Meeting on Foundations of Software Engineering, ESEC/FSE’17, pp 129–139. Association for Computing Machinery, New York

Gopstein D, Zhou HH, Frankl P, Cappos J (2018) Prevalence of Confusing Code in Software Projects: Atoms of Confusion in the Wild. In: Proceedings of the International Conference on Mining Software Repositories, ICSMR’18, pp 281–291. Association for Computing Machinery, New York

Holmqvist K, Nyström M, Andersson R, Dewhurst R, Jarodzka H, Van de Weijer J (2011) Eye Tracking: A Comprehensive Guide to Methods and Measures. OUP Oxford

Jadhav A, Pramod D, Ramanathan K (2019) Comparison of Performance of Data Imputation Methods for Numeric Dataset. Appl Artif Intell 33(10):913–933

Just MA, Carpenter PA (1980) A Theory of Reading: From Eye Fixations to Comprehension. Psychol Rev 87(4):329

Kleinke K (2017) Multiple Imputation under Violated Distributional Assumptions: A Systematic Evaluation of the Assumed Robustness of Predictive Mean Matching. J Educ Behav Stat 42(4):371–404

Langhout C, Aniche M (2021) Atoms of Confusion in Java. In: Proceedings of the International Conference on Program Comprehension, ICPC’21, IEEE, pp 25–35

Lawrie D, Feild H, Binkley D (2007) Quantifying Identifier Quality: an Analysis of Trends. Empir Softw Eng 12(4):359–388

Malaquias R, Ribeiro M, Bonifácio R, Monteiro E, Medeiros F, Garcia A, Gheyi R (2017) The Discipline of Preprocessor-Based Annotations – Does #ifdef TAG n’t #endif Matter. In: Proceedings of the International Conference on Program Comprehension, ICPC’17, IEEE, pp 297–307

Medeiros F, Lima G, Amaral G, Apel S, Kästner C, Ribeiro M, Gheyi R (2019) An Investigation of Misunderstanding Code Patterns in C Open-source Software Projects. Empir Softw Eng 24(4):1693–1726

Melo J, Narcizo FB, Hansen DW, Brabrand C, Wasowski A (2017) Variability Through the Eyes of the Programmer. In: Proceedings of the International Conference on Program Comprehension, ICPC’17, IEEE Press, pp 34–44

Mendes W, Viana W, Rocha L (2021) BOHR - Uma Ferramenta para a Identificação de Átomos de Confusão em Códigos Java. In: Workshop de Visualização, Evolução e Manutenção de Software, VEM’21, SBC, pp 41–45. Sociedade Brasileira de Computação

Nyström M, Holmqvist K (2010) An Adaptive Algorithm for Fixation, Saccade, and Glissade Detection in Eyetracking Data. Behav Res Methods 42(1):188–204

Obaidellah U, Al Haek M, Cheng PC-H (2018) A Survey on the Usage of Eye-tracking in Computer Programming. ACM Comput Surv (CSUR) 51(1):1–58

Oliveira D, Bruno R, Madeiral F, Castor F (2020) Evaluating Code Readability and Legibility: An Examination of Human-centric Studies. In: Proceedings of the International Conference on Software Maintenance and Evolution, ICSME’20. Association for Computing Machinery, New York

Rayner K (1998) Eye Movements in Reading and Information Processing: 20 Years of Research. Psychol Bull 124(3):372

Sadun E, Gregor D (2015) Remove c-style for-loops with conditions and incrementers. Swift Programming Language Evolution, proposal SE-0007. https://github.com/apple/swift-evolution/blob/main/proposals/0007-remove-c-style-for-loops.md. Accessed 21 March 2023

Salvucci D, Goldberg J (2000) Identifying Fixations and Saccades in Eye-tracking Protocols. In: Proceedings of the Symposium on Eye Tracking Research & Applications, ETRA’00, pp 71–78. Association for Computing Machinery, New York

Schröter I, Krüger J, Siegmund J, Leich T (2017) Comprehending Studies on Program Comprehension. In: Proceedings of the International Conference on Program Comprehension, ICPC’20, IEEE, pp 308–311

Schulze S, Liebig J, Siegmund J, Apel S (2013) Does the Discipline of Preprocessor Annotations Matter?: A Controlled Experiment. In: Proceedings of the International Conference on Generative Programming: Concepts & Experiences, GPCE ’13, pp 65–74. Association for Computing Machinery, New York

Shapiro SS, Wilk MB (1965) An analysis of variance test for normality (complete samples). Biometrika 52(3/4):591–611

Sharafi Z, Shaffer T, Sharif B, Guéhéneuc Y-G (2015) Eye-tracking Metrics in Software Engineering. In: Proceedings of the Asia-Pacific Software Engineering Conference, APSEC’15, IEEE, pp 96–103

Sharafi Z, Sharif B, Guéhéneuc Y-G, Begel A, Bednarik R, Crosby M (2020) A Practical Guide on Conducting Eye Tracking Studies in Software Engineering. Empir Softw Eng 25(5):3128–3174

Sharafi Z, Soh Z, Guéhéneuc Y-G, Antoniol G (2012) Women and Men-Different but Equal: On the Impact of Identifier Style on Source Code Reading. In: Proceedings of the International Conference on Program Comprehension, ICPC’12, IEEE, pp 27–36

Sharif B, Falcone M, Maletic J (2012) An Eye-tracking Study on the Role of Scan Time in Finding Source Code Defects. In: Proceedings of the Symposium on Eye Tracking Research & Applications, ETRA’12, ACM, pp 381–384

Sharif B, Maletic J (2010) An Eye Tracking Study on Camelcase and Under_score Identifier Styles. In: Proceedings of the International Conference on Program Comprehension, ICPC’10, IEEE, pp 196–205

Sheskin DJ (2020) Handbook of Parametric and Nonparametric Statistical Procedures. CRC Press

Stefik A, Siebert S (2013) An Empirical Investigation into Programming Language Syntax. Trans Comput Educ 13(4):1–40

Strauss A, Corbin J (1998) Basics of Qualitative Research Techniques. Citeseer

Yeh MK-C, Yan Y, Zhuang Y, DeLong LA (2021) Identifying Program Confusion Using Electroencephalogram Measurements. Behav Inf Technol 41:1–18

Thông tin
Thông tin xuất bản

Empirical Software Engineering

Tập 28

1-42

Thông tin tác giả