How developers engage with static analysis tools in different contexts
Tóm tắt
Automatic static analysis tools (ASATs) are instruments that support code quality assessment by automatically detecting defects and design issues. Despite their popularity, they are characterized by (i) a high false positive rate and (ii) the low comprehensibility of the generated warnings. However, no prior studies have investigated the usage of ASATs in different development contexts (e.g., code reviews, regular development), nor how open source projects integrate ASATs into their workflows. These perspectives are paramount to improve the prioritization of the identified warnings. To shed light on the actual ASATs usage practices, in this paper we first survey 56 developers (66% from industry and 34% from open source projects) and interview 11 industrial experts leveraging ASATs in their workflow with the aim of understanding how they use ASATs in different contexts. Furthermore, to investigate how ASATs are being used in the workflows of open source projects, we manually inspect the contribution guidelines of 176 open-source systems and extract the ASATs’ configuration and build files from their corresponding GitHub repositories. Our study highlights that (i) 71% of developers do pay attention to different warning categories depending on the development context; (ii) 63% of our respondents rely on specific factors (e.g., team policies and composition) when prioritizing warnings to fix during their programming; and (iii) 66% of the projects define how to use specific ASATs, but only 37% enforce their usage for new contributions. The perceived relevance of ASATs varies between different projects and domains, which is a sign that ASATs use is still not a common practice. In conclusion, this study confirms previous findings on the unwillingness of developers to configure ASATs and it emphasizes the necessity to improve existing strategies for the selection and prioritization of ASATs warnings that are shown to developers.
Tài liệu tham khảo
Hovemeyer D, Pugh W (2004) Finding Bugs is Easy. In: OOPSLA 2004, ACM, pp 132–136. http://doi.acm.org/10.1145/1028664.1028717
Al Shalabi L, Shaaban Z, Kasasbeh B (2006) Data mining: a preprocessing engine. J Comput Sci 2(9):735–739
Ayewah N, Pugh W, Hovemeyer D, Morgenthaler JD, Penix J (2008) Using static analysis to find bugs. IEEE Softw 25(5):22–29. https://doi.org/10.1109/MS.2008.130
Ayewah N, Pugh W, Morgenthaler JD, Penix J, Zhou Y (2007) Evaluating static analysis defect warnings on production software. In: Das M, Grossman D (eds) Proceedings of the 7th ACM SIGPLAN-SIGSOFT workshop on program analysis for software tools and engineering, PASTE’07, San Diego, California, USA, June 13-14, 2007. https://doi.org/10.1145/1251535.1251536. ACM, pp 1–8
Bacchelli A, Bird C (2013) Expectations, outcomes, and challenges of modern code review. In: Proceedings of the 2013 international conference on software engineering. IEEE Press, pp 712–721
Balachandran V (2013) Reducing human effort and improving quality in peer code reviews using automatic static analysis and reviewer recommendation. In: Proceedings of the international conference on software engineering (ICSE). https://doi.org/10.1109/ICSE.2013.6606642. IEEE, pp 931–940
Beller M, Bacchelli A, Zaidman A, Juergens E (2014) Modern code reviews in open-source projects: which problems do they fix?. In: Proceedings of the 11th working conference on mining software repositories. ACM, pp 202–211
Beller M, Bholanath R, McIntosh S, Zaidman A (2016) Analyzing the state of static analysis: a large-scale evaluation in open source software. In: IEEE 23rd international conference on software analysis, evolution, and reengineering, SANER 2016, Suita, Osaka, Japan, March 14-18, 2016. https://doi.org/10.1109/SANER.2016.105, vol 1. IEEE Computer Society, pp 470–481
Beller M, Gousios G, Panichella A, Proksch S, Amann S, Zaidman A (2017) Developer testing in the IDE: patterns, beliefs, and behavior. IEEE Trans Softw Eng (TSE)
Beller M, Gousios G, Panichella A, Zaidman A (2015a) When, how and why developers (do not) test in their IDEs. In: Proceedings of the joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering (ESEC/FSE). ACM, pp 179–190
Beller M, Gousios G, Zaidman A (2015b) How (much) do developers test?. In: 37th IEEE/ACM international conference on software engineering (ICSE 2015). IEEE Computer Society, pp 559–562
Beller M, Gousios G, Zaidman A (2017) Oops, my tests broke the build: an explorative analysis of travis ci with github. In: International conference on mining software repositories. IEEE Press, pp 356–367
Beller M, Gousios G, Zaidman A (2017) TravisTorrent: synthesizing Travis CI and GitHub for full-stack research on continuous integration. In: Proceedings of the 14th working conference on mining software repositories. IEEE, pp 447–450
Bitbucket (2019) https://bitbucket.org/. Accessed: 2019-03-10
Bodden E (2018) Self-adaptive static analysis. In: Proceedings of the 40th international conference on software engineering: new ideas and emerging results, ICSE-NIER ’18. https://doi.org/10.1145/3183399.3183401. ACM, New York, pp 45–48
Buckers T, Cao C, Doesburg M, Gong B, Wang S, Beller M, Zaidman A (2017) UAV: warnings from multiple automated static analysis tools at a glance. In: IEEE 24th international conference on software analysis, evolution and reengineering (SANER). IEEE Computer Society, pp 472–476
Bundler (2019) https://bundler.io/. Accessed: 2019-03-10
Butler S, Wermelinger M, Yu Y, Sharp H (2010) Exploring the influence of identifier names on code quality: an empirical study. In: Proceedings of the European conference on software maintenance and reengineering (CSMR), pp 156–165
Catolino G, Palomba F, De Lucia A, Ferrucci F, Zaidman A (2018) Enhancing change prediction models using developer-related factors. J Syst Softw 143:14–28
Checkmarx (2019) https://www.checkmarx.com/. Accessed: 2019-03-10
CheckStyle (2019) http://checkstyle.sourceforge.net. Accessed: 2019-03-10
Chen L (2015) Continuous delivery: huge benefits, but challenges too. IEEE Softw 32(2):50–54
Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37–46
CodePro (2019) https://www.roseindia.net/eclipse/plugins/tool/CodePro-AnalytiX.shtml. Accessed: 2019-03-10
CryptLife (2017) Top ten forums for programmers, https://www.cryptlife.com/designing/programming/10-best-active-forums-for-programmers
Di Penta M, Cerulo L, Aversano L (2009) The life and death of statically detected vulnerabilities: an empirical study. Inf Softw Technol 51(10):1469–1484
Dias M, Cassou D, Ducasse S (2013) Representing code history with development environment events. In: International workshop on smalltalk technologies
Dillman DA, Smyth JD, Christian LM (2014) Internet, phone, mail, and mixed-mode surveys: the tailored design method. Wiley, New York
D’silva V, Kroening D, Weissenbacher G (2008) A survey of automated techniques for formal software verification. IEEE Trans Comput Aided Des Integr Circuits Syst 27(7):1165–1178
Emanuelsson P, Nilsson U (2008) A comparative study of industrial static analysis tools. Electron Notes Theor Comput Sci 217:5–21
ESLint (2019) https://eslint.org/. Accessed: 2019-03-10
Findbugs (2019) http://findbugs.sourceforge.net/index.html. Accessed: 2019-03-10
flake8 (2019) http://flake8.pycqa.org/en/latest/. Accessed: 2019-03-10
Flanagan C, Leino KRM, Lillibridge M, Nelson G, Saxe JB, Stata R (2002) Extended static checking for java. In: Proceedings of the ACM SIGPLAN conference on programming language design and implementation (PLDI), pp 234–245
Flow (2019) https://flow.org/ Accessed: 2019-03-10
Gibbs L, Kealy M, Willis K, Green J, Welch N, Daly J (2007) What have sampling and data collection got to do with good qualitative research? Aust N Z J Public Health 31(6):540–544
Gerrit (2019) https://code.google.com/p/gerrit/. Accessed: 2019-03-10
Github (2019) https://github.com/. Accessed: 2019-03-10
Gitlab (2019) https://about.gitlab.com/. Accessed: 2019-03-10
Gousios G, Pinzger M, van Deursen A (2014) An exploratory study of the pull-based software development model. In: Proceedings of the 36th international conference on software engineering, ICSE 2014. https://doi.org/10.1145/2568225.2568260. ACM, New York, pp 345–355
Gousios G, Zaidman A, Storey MA, van Deursen A (2015) Work practices and challenges in pull-based development: the integrator’s perspective. In: 37th IEEE/ACM international conference on software engineering (ICSE 2015). IEEE computer society, pp 358–368
Gradle (2019) https://gradle.org/. Accessed: 2019-03-10
Heckman SS, Williams LA (2011) A systematic literature review of actionable alert identification techniques for automated static code analysis. Inf Softw Technol 53 (4):363–387. https://doi.org/10.1016/j.infsof.2010.12.007
Hilton M, Tunnell T, Huang K, Marinov D, Dig D (2016) Usage, costs, and benefits of continuous integration in open-source projects. In: Proceedings of the 31st IEEE/ACM international conference on automated software engineering (ASE 2016). ACM, pp 426–437
Coverity (2009) Effective management of static analysis vulnerabilities and defects. https://pdfs.semanticscholar.org/1970/a4d1746734577a6eb4fdd783668f6b4202ef.pdf. Accessed 20 Aug 2019
Johnson B, Song Y, Murphy-Hill ER, Bowdidge RW (2013) Why don’t software developers use static analysis tools to find bugs? In: Notkin D, Cheng BHC, Pohl K (eds) 35th international conference on software engineering, ICSE’13, San Francisco, CA, USA, May 18-26, 2013. https://doi.org/10.1109/ICSE.2013.6606613. IEEE Computer Society, pp 672–681
Johnson RB, Onwuegbuzie AJ (2004) Mixed methods research: a research paradigm whose time has come. Educ Res 33(7):14–26
Johnson SC (1977) Lint, a C program checker. Bell Telephone Laboratories Murray Hill
Jørgensen M (2004) A review of studies on expert estimation of software development effort. J Syst Softw 70(1-2):37–60
JSHint (2019) https://jshint.com/. Accessed: 2019-03-10
Khoshgoftaar TM, Allen EB (1998) Classification of fault-prone software modules: prior probabilities, costs, and model evaluation. Empir Softw Eng 3(3):275–298
Kim S, Ernst MD (2007) Which warnings should I fix first?. In: Proceedings of the the 6th joint meeting of the european software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering, ESEC-FSE ’07. https://doi.org/10.1145/1287624.1287633. ACM, pp 45–54
Krippendorff K (2004) Content analysis: an introduction to its methodology, 2nd edn. Sage, London
Lehman MM (1980) On understanding laws, evolution, and conservation in the large-program life cycle. J Syst Softw 1:213–221. https://doi.org/10.1016/0164-1212(79)90022-0
Mahmood R, Mahmoud QH (2018) Evaluation of static analysis tools for finding vulnerabilities in java and C/C++ source code. arXiv:1805.09040
Maven (2019) http://maven.apache.org/plugins/index.html Accessed: 2019-03-10
McIntosh S, Kamei Y, Adams B, Hassan AE (2014) The impact of code review coverage and code review participation on software quality: a case study of the qt, vtk, and ITK projects. In: Proceedings of the working conference on mining software repositories (MSR), pp 192–201
Muske T, Talluri R, Serebrenik A (2018) Repositioning of static analysis alarms. In: Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis, ISSTA 2018. https://doi.org/10.1145/3213846.3213850. ACM, New York, pp 187–197
Nagappan N, Ball T (2005) Static analysis tools as early indicators of pre-release defect density. In: Proceedings of the international conference on software engineering (ICSE), pp 580–586
Nanda MG, Gupta M, Sinha S, Chandra S, Schmidt D, Balachandran P (2010) Making defect-finding tools work for you. In: Proceedings of the international conference on software engineering (ASE), vol 2, pp 99–108
Novak J, Krajnc A, žontar R (2010) Taxonomy of static code analysis tools. In: The 33rd international convention MIPRO, pp 418–422
Nurolahzade M, Nasehi SM, Khandkar SH, Rawal S (2009) The role of patch review in software evolution: an analysis of the mozilla firefox. In: Proceedings of the joint international and annual ERCIM workshops on principles of software evolution (IWPSE) and software evolution (Evol) workshops, pp 9–18
Oppenheim B (1992) Questionnaire design, interviewing and attitude measurement. Pinter Publishers, London
Palomba F, Zanoni M, Fontana FA, De Lucia A, Oliveto R (2017) Toward a smell-aware bug prediction model. IEEE Trans Softw Eng 45(2):194–218
Panichella S, Arnaoudova V, Di Penta M, Antoniol G (2015) Would static analysis tools help developers with code reviews?. In: 22nd IEEE international conference on software analysis, evolution, and reengineering, SANER 2015, Montreal, QC, Canada, March 2-6, 2015. https://doi.org/10.1109/SANER.2015.7081826, pp 161–170
Parnas DL, Lawford M (2003) The role of inspection in software quality assurance. IEEE Trans Softw Eng 29(8):674–676
PEP8 online check (2019) http://pep8online.com/ Accessed: 2019-03-10
PMD (2019) http://pmd.sourceforge.net. Accessed: 2019-03-10
Prettier (2019) https://prettier.io/ Accessed: 2019-03-10
Proksch S, Nadi S, Amann S, Mezini M (2017) Enriching in-ide process information with fine-grained source code history. In: International conference on software analysis, evolution, and reengineering
Pylint (2019) https://www.pylint.org/. Accessed: 2019-03-10
Rahman F, Khatri S, Barr ET, Devanbu PT (2014) Comparing static bug finders and statistical prediction. In: Proceedings of the international conference on software engineering (ICSE), pp 424–434
Reddit (2017a) Php static analysis tools, https://www.reddit.com/r/PHP/comments/5d4ptt/static_code_analysis_tools_veracode/
Reddit (2017b) Static analysis tools, https://www.reddit.com/r/programming/comments/3087rz/static_code_analysis/
Reddit (2019) https://www.reddit.com/. Accessed: 2019-03-10
Rigby PC (2011) Understanding open source software peer review: review processes, parameters and statistical models, and underlying behaviours and mechanisms. Ph.D. thesis, University of Victoria, BC Canada
Rigby PC, German DM (2006) A preliminary examination of code review processes in open source projects. Tech. Rep. DCS-305-IR, University of Victoria
Runeson P, Höst M (2009) Guidelines for conducting and reporting case study research in software engineering. Empirical Softw Engg 14(2):131–164. https://doi.org/10.1007/s10664-008-9102-8
Rubocop (2019) https://github.com/rubocop-hq/rubocop Accessed: 2019-03-10
Ruthruff JR, Penix J, Morgenthaler JD, Elbaum S, Rothermel G (2008) Predicting accurate and actionable static analysis warnings: an experimental approach. In: Proceedings of the 30th international conference on software engineering. ACM, pp 341–350
Sadowski C, Aftandilian E, Eagle A, Miller-Cushon L, Jaspan C (2018) Lessons from building static analysis tools at Google. Commun ACM 61(4):58–66. https://doi.org/10.1145/3188720
Sadowski C, van Gogh J, Jaspan C, Söderberg E, Winter C (2015) Tricorder: building a program analysis ecosystem. In: Bertolino A, Canfora G, Elbaum SG (eds) 37th IEEE/ACM international conference on software engineering, ICSE 2015, Florence, Italy, May 16-24, 2015. https://doi.org/10.1109/ICSE.2015.76, vol 1. IEEE Computer Society, pp 598–608
SBT (2019) https://www.scala-sbt.org/. Accessed: 2019-03-10
SonarQube (2019) http://www.sonarqube.org Accessed: 2019-03-10
Spencer D (2009) Card sorting: designing usable categories. Rosenfeld Media
StackOverflow (2017a) Static analysis tool customatization, https://stackoverflow.com/questions/2825261/static-analysis-tool-customization-for-any-language
StackOverflow (2017b) Static analysis tools, https://stackoverflow.com/questions/22617713/whats-the-current-state-of-static-analysis-tools-for-scala
Thung F, Lucia L, Lo D, Jiang L, Rahman F, Devanbu PT (2012) To what extent could we detect field defects? an empirical study of false negatives in static bug finding tools. In: Proceedings of the international conference on automated software engineering (ASE), pp 50–59
Vassallo C, Palomba F, Bacchelli A, Gall HC (2018) Continuous code quality: are we (really) doing that?. In: ASE. ACM, pp 790–795
Vassallo C, Palomba F, Gall HC (2018) Continuous refactoring in ci: a preliminary study on the perceived advantages and barriers. In: 34th IEEE international conference on software maintenance and evolution (ICSME)
Vassallo C, Panichella S, Palomba F, Proksch S, Gall HC, Zaidman A (2019) Replication package for “How developers engage with static analysis tools in different contexts”. https://doi.org/10.5281/zenodo.3253223
Vassallo C, Panichella S, Palomba F, Proksch S, Zaidman A, Gall HC (2018) Context is king: the developer perspective on the usage of static analysis tools. In: SANER. IEEE Computer Society, pp 38–49
Vassallo C, Proksch S, Zemp T, Gall HC (2018) Un-break my build: assisting developers with build repair hints. In: International conference on program comprehension (ICPC). IEEE
Vassallo C, Schermann G, Zampetti F, Romano D, Leitner P, Zaidman A, Di Penta M, Panichella S (2017) A tale of CI build failures: an open source and a financial organization perspective. In: 2017 IEEE international conference on software maintenance and evolution, ICSME 2017, Shanghai, China, September 17-22, 2017. https://doi.org/10.1109/ICSME.2017.67. IEEE Computer Society, pp 183–193
Vassallo C, Zampetti F, Romano D, Beller M, Panichella A, Di Penta M, Zaidman A (2016) Continuous delivery practices in a large financial organization. In: 32nd IEEE international conference on software maintenance and evolution (ICSME), pp 41–50
Wagner S, Jürjens J, Koller C, Trischberger P (2005) Comparing bug finding tools with reviews and tests. In: Proceedings of the 17th IFIP TC6/WG 6.1 international conference on testing of communicating systems, pp 40–55
Wikis (2019) https://help.github.com/en/articles/about-wikis Accessed: 2019-03-10
Zampetti F, Scalabrino S, Oliveto R, Canfora G, Di Penta M (2017) How open source projects use static code analysis tools in continuous integration pipelines. In: Proceedings of the 14th international conference on mining software repositories. IEEE Press, pp 334–344
Zheng J, Williams L, Nagappan N, Snipes W, Hudepohl J, Vouk M (2006) On the value of static analysis for fault detection in software. IEEE Trans Softw Eng (TSE) 32(4):240–253