Tôi có thực sự cần tất cả công việc này để tìm ra các lỗ hổng không?

Empirical Software Engineering - Tập 27 - Trang 1-78 - 2022
Sarah Elder1, Nusrat Zahan1, Rui Shu1, Monica Metro1, Valeri Kozarev1, Tim Menzies1, Laurie Williams1
1North Carolina State University (NCSU) Department of Computer Science, College of Engineering, Raleigh, USA

Tóm tắt

Việc áp dụng các kỹ thuật phát hiện lỗ hổng là một trong nhiều nhiệm vụ sử dụng nguồn lực hạn chế của một dự án phần mềm. Mục tiêu của nghiên cứu này là hỗ trợ các nhà quản lý và những người ra quyết định khác trong việc đưa ra những lựa chọn thông minh về việc sử dụng các kỹ thuật phát hiện lỗ hổng phần mềm thông qua một nghiên cứu thực nghiệm về hiệu quả và tính hiệu suất của bốn kỹ thuật trên một ứng dụng web dựa trên Java. Chúng tôi áp dụng bốn loại kỹ thuật phát hiện lỗ hổng khác nhau – kiểm thử xâm nhập thủ công có hệ thống (SMPT), kiểm thử xâm nhập thủ công khám phá (EMPT), kiểm thử bảo mật ứng dụng động (DAST) và kiểm thử bảo mật ứng dụng tĩnh (SAST) – trên một hệ thống hồ sơ y tế mã nguồn mở. Chúng tôi đã tìm thấy nhiều lỗ hổng nhất bằng cách sử dụng SAST. Tuy nhiên, EMPT phát hiện nhiều lỗ hổng nghiêm trọng hơn. Với mỗi kỹ thuật, chúng tôi đã tìm thấy các lỗ hổng duy nhất mà không thể phát hiện bằng các kỹ thuật khác. Hiệu suất của các kỹ thuật thủ công (EMPT, SMPT) tương đương hoặc tốt hơn so với hiệu suất của các kỹ thuật tự động (DAST, SAST) về mặt số lỗ hổng phát hiện được trong một giờ (VpH). Kỹ thuật phát hiện lỗ hổng mà các thực hành viên nên lựa chọn có thể thay đổi tùy thuộc vào mục tiêu và nguồn lực có sẵn của dự án. Nếu mục tiêu của một tổ chức là tìm ra "tất cả" các lỗ hổng trong một dự án, họ cần sử dụng càng nhiều kỹ thuật càng tốt trong khả năng nguồn lực của mình.

Từ khóa

#phát hiện lỗ hổng #kiểm thử xâm nhập #bảo mật ứng dụng #hiệu quả kỹ thuật #dự án phần mềm

Tài liệu tham khảo

Ackerman E (2019) Upgrade to superhuman reflexes without feeling like a robot. IEEE Spectr. https://spectrum.ieee.org/enabling-superhuman-reflexes-without-feeling-like-a-robot Alomar N, Wijesekera P, Qiu E, Egelman S (2020) “you’ve got your nice list of bugs, now what?” vulnerability discovery and management processes in the wild. In: Sixteenth Symposium on Usable Privacy and Security ({SOUPS} 2020), pp 319–339 Amankwah R, Chen J, Kudjo PK, Towey D (2020) An empirical comparison of commercial and open-source web vulnerability scanners. Softw - Pract Exp 50(9):1842–1857 Anderson T (2020) Linux in 2020: 27.8 million lines of code in the kernel, 1.3 million in systemd. The Register URL https://www.theregister.com/2020/01/06/linux_2020_kernel_systemd_code/. Accessed 21 Dec 2021 Antunes N, Vieira M (2009) Comparing the effectiveness of penetration testing and static code analysis on the detection of sql injection vulnerabilities in web services. In: 2009 15th IEEE Pacific Rim International Symposium on Dependable Computing. IEEE, pp 301–306 Antunes N, Vieira M (2010) Benchmarking vulnerability detection tools for web services. In: 2010 IEEE International Conference on Web Services,IEEE, pp. 203–210 Austin A, Holmgreen C, Williams L (2013) A comparison of the efficiency and effectiveness of vulnerability discovery techniques. Inf Softw Technol 55(7):1279–1288 Austin A, Williams L (2011) One technique is not enough: A comparison of vulnerability discovery techniques. In: 2011 International Symposium on Empirical Software Engineering and Measurement, IEEE, pp. 97–106 Bannister A (2021) Healthcare provider texas ent alerts 535,000 patients to data breach. The Daily Swig. [Online; Publication Date 20 Dec 2021; Accessed 21 Dec 2021] Bartlett MS (1937) Properties of sufficiency and statistical tests. Proc R Soc A: Math Phys Eng Sci 160(901):268–282 Bau J, Wang F, Bursztein E, Mutchler P, Mitchell JC (2012) Vulnerability factors in new web applications: Audit tools, developer selection & languages. Tech. rep., Stanford, https://seclab.stanford.edu/websec/scannerPaper.pdf. Accessed 21 Dec, 2021 Campbell GA (2020) What is ’taint analysis’ and why do i care?, https://blog.sonarsource.com/what-is-taint-analysis Cass S (2021) Top programming languages 2021. IEEE Spectr, https://spectrum.ieee.org/top-programming-languages-2021. Accessed 21 Dec 2021 Cass S, Kulkarni P, Guizzo E (2021) Interactive: Top Programming Languages 2021. IEEE Spectrum, https://spectrum.ieee.org/top-programming-languages/. Accessed 20 Apr 2022 Chaim ML, Santos DS, Cruzes DS (2018) What do we know about buffer overflow detection?: A survey on techniques to detect a persistent vulnerability. International Journal of Systems and Software Security and Protection (IJSSSP) 9(3):1–33 Cicchetti DV, Feinstein AR (1990) High agreement but low kappa: Ii. resolving the paradoxes. J Clin Epidemiol 43(6):551–558 Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37–46 Condon C, Miller H (2021) Maryland health department says there’s no evidence of data lost after cyberattack; website is back online. Baltimore Sun, https://www.baltimoresun.com/health/bs-hs-mdh-website-down-20211206-o2ky2sn5znb3pdwtnu2a7m5g6q-story.html. Accessed 21 Dec 2021 Cook TD, Campbell DT (1979) Quasi-experimentation: Design and analysis issues for field settings. Rand McNally College Publishing, Chicago Corbin J, Strauss A (2008) Basics of qualitative research: Techniques and procedures for developing grounded theory, 3rd edn. SAGE Publications Inc., California Cowan C, Wagle F, Pu C, Beattie S, Walpole J (2000) Buffer overflows: Attacks and defenses for the vulnerability of the decade. In: Proceedings DARPA Information Survivability Conference and Exposition. DISCEX’00, IEEE, vol. 2, pp. 119–129 Cruzes DS, Felderer M, Oyetoyan TD, Gander M, Pekaric I (2017) How is security testing done in agile teams? a cross-case analysis of four software teams. In: International Conference on Agile Software Development, Springer, Cham, pp. 201–216 Dambra S, Bilge L, Balzarotti D (2020) Sok: Cyber insurance–technical challenges and a system security roadmap. In: 2020 IEEE Symposium on Security and Privacy (SP), IEEE, pp. 1367–1383 Davis FD (1989) Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly 13(3):319–340 Delaitre AM, Stivalet BC, Black PE, Okun V, Cohen TS, Ribeiro A (2018) Sate v report: Ten years of static analysis tool expositions. NIST SP 500-326, National Institute of Standards and Technology (NIST), https://doi.org/10.6028/NIST.SP.500-326. Accessed 20 Jul 2021 Desjardins J (2017) Here’s how many millions of lines of code it takes to run different software. Business Insider, https://www.businessinsider.com/how-many-lines-of-code-it-takes-to-run-different-software-2017-2. Accessed 21 Dec 2021 Doupé A, Cova M, Vigna G (2010) Why johnny can’t pentest: An analysis of black-box web vulnerability scanners. In: International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment, Springer, pp. 111–131 Elder SE, Zahan N, Kozarev V, Shu R, Menzies T, Williams L (2021) Structuring a comprehensive software security course around the owasp application security verification standard. In: 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET), IEEE, pp. 95–104 Epic Systems Corporation (2020) From healthcare to mapping the milky way: 5 things you didn’t know about epic’s tech, https://www.epic.com/epic/post/healthcare-mapping-milky-way-5-things-didnt-know-epics-tech. Accessed 07 Dec 2021 Executive Order 14028 (2021) Executive order on improving the nation’s cybersecurity. Exec. Order No. 14028, 86 FR 26633, https://www.federalregister.gov/d/2021-10460 Feinstein AR, Cicchetti DV (1990) High agreement but low kappa: I. the problems of two paradoxes. J Clin Epidemiol 43(6):543–549 Feldt R, Magazinius A (2010) Validity threats in empirical software engineering research-an initial survey.. In: Seke, pp. 374–379 Feng GC (2013) Factors affecting intercoder reliability: A monte carlo experiment. Qual Quant 47(5):2959–2982 Fielding RT, Reschke J (2014) Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content. RFC Editor https://doi.org/10.7231/RFC7231, https://rfc-editor.org/rfc/rfc7231.txt. Accessed 21 Dec 2021 Finifter M, Akhawe D, Wagner D (2013) An empirical study of vulnerability rewards programs. In: 22nd USENIX Security Symposium (USENIX Security 13), USENIX, Washington, D.C., pp. 273–288 Fonseca J, Vieira M, Madeira H (2007) Testing and comparing web vulnerability scanning tools for sql injection and xss attacks. In: 13th Pacific Rim international symposium on dependable computing (PRDC 2007), IEEE, pp. 365–372 Games PA, Howell JF (1976) Pairwise multiple comparison procedures with unequal n’s and/or variances: a monte carlo study. J Educ Stat 1(2):113–125 Github (2021) The 2021 State of the Octoverse, https://octoverse.github.com/. Accessed 20 Apr 2022 Gonçales L, Farias K, da Silva BC (2021) Measuring the cognitive load of software developers: An extended systematic mapping study. Inf Softw Technol 106563 Hafiz M, Fang M (2016) Game of detections: how are security vulnerabilities discovered in the wild?. Empir Softw Eng 21(5):1920–1959 Imtiaz N, Rahman A, Farhana E, Williams L (2019) Challenges with responding to static analysis tool alerts. In: 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR), IEEE, pp. 245–249 ISO/IEC/IEEE (2013) Software and systems engineering — software testing — part 1: concepts and definitions. ISO/IEC/IEEE 29119-1:2013, International Organization for Standardization (ISO), International Electrotechnical Commission (IES), and Institute of Electrical and Electronics Engineers (IEEE) Itkonen J, Mäntylä MV (2014) Are test cases needed? replicated comparison between exploratory and test-case-based software testing. Empir Softw Eng 19(2):303–342 Itkonen J, Mäntylä MV, Lassenius C (2013) The role of the tester’s knowledge in exploratory software testing. IEEE Trans Softw Eng 39(5):707–724 Johnson B, Song Y, Murphy-Hill E, Bowdidge R (2013) Why don’t software developers use static analysis tools to find bugs?. In: Proceedings of the 2013 International Conference on Software Engineering, IEEE Press, pp. 672–681 Joint Task Force Transformation Initiative (2013) Security and privacy controls for federal information systems and organizations. NIST SP 800-53, National Institute of Standards and Technology (NIST), https://doi.org/10.6028/NIST.SP.800-53r4. Accessed 20 Jul 2021 Kirk R (2013) Experimental design: Procedures for the behavioral sciences, 4th edn. Sage Publications, Thousand Oaks Kitchenham B, Madeyski L, Budgen D, Keung J, Brereton P, Charters S, Gibbs S, Pohthong A (2017) Robust statistical methods for empirical software engineering. Empir Softw Eng 22(2):579–630 Kitchenham BA, Budgen D, Brereton P (2015) Evidence-based software engineering and systematic reviews, vol 4. CRC press, Boca Raton Klees G, Ruef A, Cooper B, Wei S, Hicks M (2018) Evaluating fuzz testing. In: Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pp. 2123–2138 Liu M, Zhang B, Chen W, Zhang X (2019) A survey of exploitation and detection methods of xss vulnerabilities. IEEE Access 7:182004–182016 Lombard M, Snyder-Duch J, Bracken CC (2002) Content analysis in mass communication: Assessment and reporting of intercoder reliability. Hum Commun Res 28(4):587–604 Lung J, Aranda J, Easterbrook S, Wilson G (2008) On the difficulty of replicating human subjects studies in software engineering. In: 2008 ACM/IEEE 30th International Conference on Software Engineering, pp. 191–200 Mallet F (2016) Sonaranalyzer for java: Tricky bugs are running scared. https://blog.sonarsource.com/sonaranalyzer-for-java-tricky-bugs-are-running-scared, Accessed 05 Dec 2021 McGraw G (2006) Software security: building security in. Addison-Wesley Professional, Boston MITRE (2016) Common vulnerabilities and exposures (cve) numbering authority (cna) rules. https://cve.mitre.org/cve/cna/CNA_Rules_v1.1.pdf, Accessed 24 July 2021 MITRE (2021) Cve → cwe mapping guidance. In: (MITRE 2021b), https://cwe.mitre.org/documents/cwe_usage/guidance.html. Accessed 24 Jul 2021 MITRE (2021) Cwe common weakness enumeration (website), https://cwe.mitre.org/. Accessed 20 Jul 2021 MITRE (2021) Cwe view: Weaknesses in owasp top ten (2021). In: (MITRE 2021b), https://cwe.mitre.org/data/definitions/1344.html. Accessed 09 Dec 2021 MITRE (2022) Cwe 1003 - cwe view: Weaknesses for simplified mapping of published vulnerabilities Morrison P, Moye D, Pandita R, Williams L (2018) Mapping the field of software life cycle security metrics. Inf Softw Technol 102:146–159 Mozilla (2021) Http messages, https://developer.mozilla.org/en-US/docs/Web/HTTP/Messages. Accessed 21 Dec 2021 Nagarakatte S, Zhao J, Martin Milo MK, Zdancewic S (2009) Softbound: Highly compatible and complete spatial memory safety for c. In: Proceedings of the 30th ACM SIGPLAN Conference on Programming Language Design and Implementation, pp 245–258 NVD (2021) Cwe over time. In: (NVD 2021b), https://nvd.nist.gov/general/visualizations/vulnerability-visualizations/cwe-over-time. Accessed 05 Dec 2021 NVD (2021) National vulnerability database (website). National Institute of Standards and Technology (NIST), https://nvd.nist.gov/. Accessed 01 Nov 2021 NVD (2021) Nvd - general faqs. National Institute of Standards and Technology (NIST). In: (NVD 2021b) https://nvd.nist.gov/general/FAQ-Sections/General-FAQs. Accessed 04 Apr 2022 NVD (2021) Vulnerabilities. In: (NVD 2021b), https://nvd.nist.gov/vuln. Accessed 01 Nov 2021 Okun V, Delaitre A, Black PE (2010) The second static analysis tool exposition (sate) 2009. NIST SP 500-287, National Institute of Standards and Technology (NIST), https://doi.org/10.6028/NIST.SP.500-287. Accessed 20 Jul 2021 Okun V, Delaitre A, Black PE (2011) Report on the third static analysis tool exposition (sate 2010). NIST SP 500-283, National Institute of Standards and Technology (NIST), https://doi.org/10.6028/NIST.SP.500-283. Accessed 20 Jul 2021 Okun V, Delaitre A, Black PE (2013) Report on the static analysis tool exposition (sate) iv. NIST SP 500-297, National Institute of Standards and Technology (NIST), https://doi.org/10.6028/NIST.SP.500-297. Accessed 20 Jul 2021 Okun V, Gaucher R, Black PE (2009) Static analysis tool exposition (sate) 2008. NIST SP 500-279, National Institute of Standards and Technology (NIST), https://doi.org/10.6028/NIST.SP.500-279. Accessed 20 Jul 2021 Open Web Application Security Project (OWASP) Foundation (2013) Owasp top ten - 2010, https://owasp.org/www-pdf-archive/OWASP_Top_10_-_2010.pdf. Accessed 05 Dec 2021 Open Web Application Security Project (OWASP) Foundation (2013) Owasp top ten - 2013, https://owasp.org/www-pdf-archive/OWASP_Top_10_-_2013.pdf. Accessed 05 Dec 2021 Open Web Application Security Project (OWASP) Foundation (2017) Owasp top ten - 2017, https://owasp.org/www-project-top-ten/2017/. Accessed 05 Dec 2021 Open Web Application Security Project (OWASP) Foundation (2021) Owasp top ten - 2021, https://owasp.org/Top10/. Accessed 05 Dec 2021 Open Web Application Security Project (OWASP) Foundation (2021) The owasp top ten application security risks project, https://owasp.org/www-project-top-ten/. Accessed 09 Dec 2021 Open Web Application Security Project (OWASP) Foundation (2021) Owasp zap, https://www.zaproxy.org/. Accessed: 21-Dec-2021 OpenMRS (2020) Openmrs developer manual, http://devmanual.openmrs.org/en/. Accessed 24 Jul 2021 OpenMRSAtlas (2021) Openmrs atlas, https://atlas.openmrs.org/. Accessed 24 Jul 2021 OWASP ZAP Dev Team (2021) Getting started - features - alerts. In: (Team OZD 2021), https://www.zaproxy.org/docs/desktop/start/features/alerts/. Accessed 06 Dec 2021 OWASP ZAP Dev Team (2021) Getting started - features - spider. In: (Team OZD 2021), https://www.zaproxy.org/docs/desktop/start/features/spider/. Accessed 20 Jul 2021 Pfahl D, Yin H, Mäntylä MV, Münch J (2014) How is exploratory testing used? a state-of-the-practice survey. In: Proceedings of the 8th ACM/IEEE international symposium on empirical software engineering and measurement, ACM, p. 5 Purkayastha S, Goyal S, Phillips T, Wu H, Haakenson B, Zou X (2020) Continuous security through integration testing in an electronic health records system. In: 2020 International Conference on Software Security and Assurance (ICSSA), IEEE, pp. 26–31 Radio New Zealand (RNZ) (2021) Health ministry announces $75m to plug cybersecurity gaps, https://www.rnz.co.nz/news/national/458331/health-ministry-announces-75m-to-plug-cybersecurity-gaps. Accessed 21 Dec 2021 Rahman AAU, Helms E, Williams L, Parnin C (2015) Synthesizing continuous deployment practices used in software development. In: 2015 Agile Conference, IEEE, pp. 1–10 Ralph P, Tempero E (2018) Construct validity in software engineering research and software metrics. In: Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering 2018, pp. 13–23 Razali NM, Wah Y B, et al (2011) Power comparisons of shapiro-wilk, kolmogorov-smirnov, lilliefors and anderson-darling tests. J Stat Modelling Anal 2(1):21–33 Scandariato R, Walden J, Joosen W (2013) Static analysis versus penetration testing: A controlled experiment. In: 2013 IEEE 24th international symposium on software reliability engineering (ISSRE), IEEE, pp. 451–460 Scanlon T (2018) 10 types of application security testing tools: When and how to use them. Blog, Software Engineering Institute, Carnegie Mellon University, https://insights.sei.cmu.edu/blog/10-types-of-application-security-testing-tools-when-and-how-to-use-them. Accessed 20 Jul 2021 Smith B, Williams L (2012) On the effective use of security test patterns. In: 2012 IEEE Sixth International Conference on Software Security and Reliability, IEEE, pp. 108–117 Smith B, Williams LA (2011) Systematizing security test planning using functional requirements phrases. Tech. Rep. TR-2011-5, North Carolina State University. Dept. of Computer Science Smith J, Do LNQ, Murphy-Hill E (2020) Why can’t johnny fix vulnerabilities: A usability evaluation of static analysis tools for security. In: Sixteenth Symposium on Usable Privacy and Security ({SOUPS} 2020), USENIX, pp. 221–238 Smith J, Johnson B, Murphy-Hill E, Chu B, Lipford HR (2015) Questions developers ask while diagnosing potential security vulnerabilities with static analysis. In: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ACM, pp. 248–259 SonarSource (2019) Sonarqube documentation: Security-related rules, https://docs.sonarqube.org/8.2/user-guide/security-rules/. Accessed 06 Dec 2021 StackOverflow (2021) 2021 Developer Survey, https://insights.stackoverflow.com/survey/2021#technology-most-popular-technologies. Accessed: 07 Dec 2021 Team OZD (ed.) (2021) The owasp zed attack proxy (zap) desktop user guide Tøndel IA, Jaatun MG, Cruzes DS, Williams L (2019) Collaborative security risk estimation in agile software development. Information & Computer Security 27(4) U.S. Cybersecurity and Infrastructure Security Agency (CISA) (2021) Provide medical care is in critical condition: Analysis and stakeholder decision support to minimize further harm, https://www.cisa.gov/sites/default/files/publications/Insights_MedicalCare_FINAL-v2_0.pdf. Accessed 21 Dec 2021 US Dept of Veterans Affairs, Office of Information and Technology, Enterprise Program Management Office (2021) VA Monograph, https://www.va.gov/vdl/documents/Monograph/Monograph/VistA_Monograph_0421_REDACTED.pdf. Accessed 07 Dec 2021 van der Stock A, Cuthbert D, Manico J, Grossman J C, Burnett M (2019) Application security verification standard. Rev. 4.0.1, Open Web Application Security Project (OWASP), https://github.com/OWASP/ASVS/tree/v4.0.1/4.0. Accessed 20 Jul 2021 Votipka D, Stevens R, Redmiles E, Hu J, Mazurek M (2018) Hackers vs. testers: A comparison of software vulnerability discovery processes. In: 2018 IEEE Symposium on Security and Privacy (SP), IEEE, pp. 374–391 Wilcox RR, Keselman HJ (2003) Modern robust data analysis methods: measures of central tendency. Psychol Methods 8(3):254 Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer Science & Business Media, New York