Một phương pháp để xác minh yêu cầu hiệu suất và tạo môi trường kiểm thử
Tóm tắt
Kiểm thử dựa trên mô hình (MBT) là một phương pháp hỗ trợ thiết kế và thực hiện các trường hợp kiểm thử thông qua các mô hình xác định hành vi mong muốn của hệ thống đang được kiểm thử. Trong khi có những tổng quan tài liệu có hệ thống về MBT nói chung, thì tình hình hiện tại về việc mô hình hóa và kiểm thử các yêu cầu hiệu suất đã nhận được ít sự chú ý hơn nhiều. Do đó, chúng tôi đã tiến hành một nghiên cứu lập bản đồ có hệ thống về kiểm thử hiệu suất dựa trên mô hình. Sau đó, chúng tôi nghiên cứu các đặc tả yêu cầu phần mềm bằng ngôn ngữ tự nhiên để hiểu các yêu cầu về hiệu suất thường được xác định như thế nào và ở mức độ nào. Vì không có kỹ thuật MBT nào đã xác định được lợi ích chính của việc mô hình hóa, đó là phát hiện các lỗi trong các đặc tả yêu cầu, chúng tôi đã phát triển phương pháp Xác minh yêu cầu hiệu suất và Tạo môi trường kiểm thử (PRO-TEST). Cuối cùng, chúng tôi đã đánh giá PRO-TEST trên 149 đặc tả yêu cầu. Chúng tôi đã tìm thấy và phân tích 57 nghiên cứu chính từ nghiên cứu lập bản đồ có hệ thống và trích xuất 50 mô hình yêu cầu hiệu suất. Tuy nhiên, những mô hình đó không đạt được các mục tiêu của MBT, đó là xác minh yêu cầu, đảm bảo tính khả thi của chúng và tạo ra các trường hợp kiểm thử tối thiểu cần thiết. Chúng tôi đã phân tích 77 tài liệu Đặc tả yêu cầu phần mềm (SRS), trích xuất 149 yêu cầu hiệu suất từ những SRS đó và minh họa rằng với PRO-TEST, chúng tôi có thể mô hình hóa các yêu cầu hiệu suất, tìm ra các vấn đề trong những yêu cầu đó và phát hiện các yêu cầu còn thiếu. Chúng tôi đã phát hiện ba yêu cầu không thể định lượng, 43 yêu cầu không được định lượng và 180 tham số chưa được xác định trong 149 yêu cầu hiệu suất đã được mô hình hóa. Hơn nữa, chúng tôi đã tạo ra 96 môi trường kiểm thử từ các mô hình đó. Bằng cách mô hình hóa các yêu cầu hiệu suất với PRO-TEST, chúng tôi có thể xác định các vấn đề trong các yêu cầu liên quan đến tính mơ hồ, khả năng đo lường và độ đầy đủ của chúng. Ngoài ra, nó cho phép tạo ra các tham số cho các môi trường kiểm thử.
Từ khóa
Tài liệu tham khảo
Abbors F, Ahmad T, Truscan D, Porres I (2013) Model-based performance testing in the cloud using the mbpet tool. In: Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering, ICPE ’13. Association for Computing Machinery, pp 423–424. https://doi.org/10.1145/2479871.2479937
Abbors F, Truşcan D (2010) Approaching performance testing from a model-based testing perspective. In: 2010 second international conference on advances in system testing and validation lifecycle, pp 125–128. https://doi.org/10.1109/VALID.2010.22
Abdeen W, Chen X, Unterkalmsteiner M (2021) Model-based testing for performance requirements dataset. https://doi.org/10.5281/zenodo.5715509
Abdelgawad M, McLeod S, Andrews A, Xiao J (2017) Model-based testing of a real-time adaptive motion planning system. Adv Robot 31(22):1159–1176. https://doi.org/10.1080/01691864.2017.1396921
Al-Qutaish RE (2010) Quality models in software engineering literature: an analytical and comparative study. J Am Sci 63:166–175
Al-tekreeti M, Naik K, Abdrabou A, Zaman M, Srivastava P (2018) Test generation for performance evaluation of mobile multimedia streaming applications. In: Proceedings of the 6th international conference on model-driven engineering and software development. SCITEPRESS - Science and Technology Publications, pp 225–236. https://doi.org/10.5220/0006609302250236
Ammann P, Offutt J (2016) Introduction to software testing. Cambridge University Press. Google-Books-ID: bQtQDQAAQBAJ
Balsamo S, Di Marco A, Inverardi P, Simeoni M (2004) Model-based performance prediction in software development: a survey. IEEE Trans Softw Eng 30(5):295–310. https://doi.org/10.1109/TSE.2004.9
Bernardino M, Zorzo AF, Rodrigues EM (2016) Canopus: a domain-specific language for modeling performance testing. In: 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST), pp 157–167. https://doi.org/10.1109/ICST.2016.13
Boehm B (1984) Verifying and validating software requirements and design specifications. IEEE Softw 1(1):75–88. https://doi.org/10.1109/MS.1984.233702
Bondi AB (2012) Best practices for writing and managing performance requirements: a tutorial. In: Proceedings of the 3rd ACM/SPEC International Conference on Performance Engineering, ICPE ’12. Association for Computing Machinery, pp 1–8. https://doi.org/10.1145/2188286.2188288
Cai Z, Yu E (2002) Addressing performance requirements using a goal and scenario-oriented approach. In: Pidduck AB, Ozsu MT, Mylopoulos J, Woo CC (eds) Advanced information systems engineering. Lecture notes in computer science. Springer, Berlin, pp 706–710. https://doi.org/10.1007/3-540-47961-9_50
Camilli M, Gargantini A, Scandurra P, Bellettini C (2017) Event-based runtime verification of temporal properties using time basic petri nets. In: Barrett C, Davies M, Kahsai T (eds) NASA formal methods. Lecture notes in computer science. Springer, Berlin, pp 115–130. https://doi.org/10.1007/978-3-319-57288-8_8
Chimisliu V, Wotawa F (2011) Abstracting timing information in UML state charts via temporal ordering and LOTOS. In: Proceedings of the 6th international workshop on Automation of Software Test, AST ’11. Association for Computing Machinery, pp 8–14. https://doi.org/10.1145/1982595.1982598
Chung L, Nixon BA, Yu E, Mylopoulos J (2012) Non-functional requirements in software engineering. Springer. Google-Books-ID: MNrcBwAAQBAJ
Clements P (1997) Coming attractions in software architecture. In: Proceedings of 5th international workshop on parallel and distributed real-time systems and 3rd workshop on object-oriented real-time systems, pp 2–9. https://doi.org/10.1109/WPDRTS.1997.637857
Coallier F (2001) Software engineering—product quality—part 1: quality model. International Organization for Standardization, Geneva
Dias Neto AC, Subramanyan R, Vieira M, Travassos GH (2007) A survey on model-based testing approaches: a systematic review. In: Proceedings of the 1st ACM international workshop on empirical assessment of software engineering languages and technologies: held in conjunction with the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE) 2007, WEASELTech ’07. Association for Computing Machinery, pp 31–36. https://doi.org/10.1145/1353673.1353681
Dias-Neto AC, Travassos GH (2010) A picture from the model-based testing area: concepts, techniques, and challenges. In: Zelkowitz MV (ed) Advances in computers, vol 80. Elsevier, Amsterdam, pp 45–120. https://doi.org/10.1016/S0065-2458(10)80002-6
Dromey R (1995) A model for software product quality. IEEE Trans Softw Eng 21(2):146–162. https://doi.org/10.1109/32.345830
Eckhardt J, Vogelsang A, Femmer H, Mager P (2016) Challenging incompleteness of performance requirements by sentence patterns. In: 2016 IEEE 24th International Requirements Engineering Conference (RE), pp 46–55 . https://doi.org/10.1109/RE.2016.24. ISSN: 2332-6441
Elmendorf WR (1973) Cause-effect graphs in functional testing. IBM Poughkeepsie Laboratory
Enoiu EP, Sundmark D, Pettersson P (2013) Model-based test suite generation for function block diagrams using the UPPAAL model checker. In: 2013 IEEE sixth international conference on software testing, verification and validation workshops, pp 158–167. https://doi.org/10.1109/ICSTW.2013.27
Faedo A. Natural language requirements dataset. Institute of Information Science and Technologies. http://fmt.isti.cnr.it/nlreqdataset/. Accessed 8 Feb 2019
Felderer M, Zech P, Breu R, Büchler M, Pretschner A (2016) Model-based security testing: a taxonomy and systematic classification. Softw Test Verif Reliab 26(2):119–148. https://doi.org/10.1002/stvr.1580
Ferrari A, Spagnolo GO, Gnesi S (2017) Towards a dataset for natural language requirements processing. In: 23rd international workshop on Requirements Engineering Foundation for Software Quality Workshops (REFSQ), p 6
Gambi A, Filieri A, Dustdar S (2013) Iterative test suites refinement for elastic computing systems. In: Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2013. Association for Computing Machinery, pp 635–638. https://doi.org/10.1145/2491411.2494579
Gangadharan D, Chakraborty S, Zimmermann R (2009) Fast model-based test case classification for performance analysis of multimedia MPSoC platforms. In: Proceedings of the 7th IEEE/ACM international conference on Hardware/software codesign and system synthesis, CODES+ISSS ’09. Association for Computing Machinery, pp 413–422. https://doi.org/10.1145/1629435.1629492
Garousi V (2011) Fault-driven stress testing of distributed real-time software based on UML models. Softw Test Verif Reliab 21(2):101–124. https://doi.org/10.1002/stvr.418
Garousi V, Zhi J (2013) A survey of software testing practices in Canada. J Syst Softw 86(5):1354–1376. https://doi.org/10.1016/j.jss.2012.12.051
Grady RB, Caswell DL (1987) Software metrics: establishing a company-wide program. Prentice-Hall, Englewood Cliffs
Hasling B, Goetz H, Beetz K (2008) Model based testing of system requirements using UML use case models. In: And validation 2008 1st international conference on software testing, verification, pp 367–376. https://doi.org/10.1109/ICST.2008.9. ISSN: 2159-4848
Ho CW, Johnson M, Williams L, Maximilien E (2006) On agile performance requirements specification and testing. In: AGILE 2006 (AGILE’06), pp 6–52. https://doi.org/10.1109/AGILE.2006.41
Hooda RV (2013) A future approach for model-based testing: issues and guidelines. Int J Latest Res Sci Technol 2(1):541–543
Häser F, Felderer M, Breu R (2014) Software paradigms, assessment types and non-functional requirements in model-based integration testing: a systematic literature review. In: Proceedings of the 18th international conference on Evaluation and Assessment in Software Engineering—EASE ’14. ACM Press, pp 1–10. https://doi.org/10.1145/2601248.2601257
Iqbal MZ, Arcuri A, Briand L (2015) Environment modeling and simulation for automated testing of soft real-time embedded software. Softw Syst Model 14(1):483–524. https://doi.org/10.1007/s10270-013-0328-6
ISO: Software product quality model—iso25010. https://iso25000.com/index.php/en/iso-25000-standards/iso-25010. Accessed 12 Sept 2019
Iyenghar P, Spieker M, Tecker P, Wuebbelmann J, Westerkamp C, van der Heiden W, Willert A (2011) Applicability of an integrated model-based testing approach for rtes. In: 2011 9th IEEE International Conference on Industrial Informatics. IEEE, pp 871–876
Johnsen A, Lundqvist K, Hänninen K, Pettersson P, Torelm M (2017) Experience report: evaluating fault detection effectiveness and resource efficiency of the architecture quality assurance framework and tool. In: 2017 IEEE 28th International Symposium on Software Reliability Engineering (ISSRE), pp 271–281. https://doi.org/10.1109/ISSRE.2017.31. ISSN: 2332-6549
Khosravi K, Guéhéneuc YG (2004) A quality model for design patterns. German Industry Standard
Kitchenham BA, Budgen D, Brereton OP (2010) The value of mapping studies—a participant-observer case study. In: 14th International Conference on Evaluation and Assessment in Software Engineering (EASE) (EASE). BCS Learning & Development. https://doi.org/10.14236/ewic/EASE2010.4
Li W, Le Gall F, Spaseski N (2018) A survey on model-based testing tools for test case generation. In: Itsykson V, Scedrov A, Zakharov V (eds) Tools and methods of program analysis, communications in computer and information science. Springer, Berlin, pp 77–89. https://doi.org/10.1007/978-3-319-71734-0_7
Li W, Le Gall F, Vlacheas P, Cheptsov A (2018) Quality assurance for component-based systems in embedded environments. In: 2018 International Conference on Internet of Things, Embedded Systems and Communications (IINTEC), pp 171–176. https://doi.org/10.1109/IINTEC.2018.8695299
Luthmann L, Stephan A, Bürdek J, Lochau M (2017) Modeling and testing product lines with unbounded parametric real-time constraints. In: Proceedings of the 21st International Systems and Software Product Line Conference—volume A, SPLC ’17. Association for Computing Machinery, pp 104–113. https://doi.org/10.1145/3106195.3106204
Löding H, Peleska J (2010) Timed moore automata: test data generation and model checking. In: Verification and validation 2010 third international conference on software testing, pp 449–458. https://doi.org/10.1109/ICST.2010.60. ISSN: 2159-4848
Maâlej AJ, Hamza M, Krichen M, Jmaïel M (2013) Automated significant load testing for WS-BPEL compositions. In: 2013 IEEE sixth international conference on software testing, verification and validation workshops, pp 144–153. https://doi.org/10.1109/ICSTW.2013.25
Maâlej AJ, Krichen M, Jmaïel M (2012) Conformance testing of WS-BPEL compositions under various load conditions. In: 2012 IEEE 36th annual computer software and applications conference, p 371. https://doi.org/10.1109/COMPSAC.2012.100. ISSN: 0730-3157
Maâlej AJ, Krichen M, Jmaïel M (2012) Model-based conformance testing of WS-BPEL compositions. In: 2012 IEEE 36th annual computer software and applications conference workshops, pp 452–457. https://doi.org/10.1109/COMPSACW.2012.86
McCall JA, Richards PK, Walters GF (1997) Factors in software quality, volumes I, II, and III. US Rome Air Development Center Reports, US Department of Commerce, USA
Molyneaux I. The art of application performance testing: from strategy to tools. O’Reilly Media, Inc. (2014-12-15). Google-Books-ID: 187UBQAAQBAJ
Moyer E. For Pokemon go, it’s stop—at least temporarily. https://www.cnet.com/news/for-pokemon-go-its-stop-at-least-temporarily/. Accessed 10 Oct 2019
Myers GJ (2004) The art of software testing, 2nd edn. Wiley, Hoboken
Nixon B (2000) Management of performance requirements for information systems. IEEE Trans Softw Eng 26(12):1122–1146. https://doi.org/10.1109/32.888627
Paradkar A, Tai K, Vouk M (1997) Specification-based testing using cause-effect graphs. Ann Softw Eng 4(1):133–157. https://doi.org/10.1023/A:1018979130614
Petersen K, Feldt R, Mujtaba S, Mattsson M (2008) Systematic mapping studies in software engineering. In: 12th international conference on Evaluation and Assessment in Software Engineering (EASE). BCS Learning & Development. https://doi.org/10.14236/ewic/EASE2008.8
Petersen K, Vakkalanka S, Kuzniarz L (2015) Guidelines for conducting systematic mapping studies in software engineering: an upyear. Inf Softw Technol 64:1–18. https://doi.org/10.1016/j.infsof.2015.03.007
Prenninger W, El-Ramly M, Horstmann M (2005) 15 case studies. In: Broy M, Jonsson B, Katoen JP, Leucker M, Pretschner A (eds) Model-based testing of reactive systems. Lecture notes in computer science, vol 3472. Springer, Berlin, pp 439–461. https://doi.org/10.1007/11498490_19
Pretschner A, Prenninger W, Wagner S, Kühnel C, Baumgartner M, Sostawa B, Zölch R, Stauner T (2005) One evaluation of model-based testing and its automation. In: Proceedings of the 27th international conference on software engineering, pp 392–401
Rodrigues E, Bernardino M, Costa L, Zorzo A, Oliveira F (2015) PLeTsPerf—a model-based performance testing tool. In: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), pp 1–8. https://doi.org/10.1109/ICST.2015.7102628. ISSN: 2159-4848
Saadatmand M, Sjödin M (2013) Testing of timing properties in real-time systems: verifying clock constraints. In: 2013 20th Asia-Pacific Software Engineering Conference (APSEC), vol 2, pp 152–158. https://doi.org/10.1109/APSEC.2013.131. ISSN: 1530-1362
Schieferdecker I (2012) Model-based testing. IEEE Softw 29(1):14–18. https://doi.org/10.1109/MS.2012.13
Schumi R, Lang P, Aichernig BK, Krenn W, Schlick R (2017) Checking response-time properties of web-service applications under stochastic user profiles. In: IFIP International Conference on Testing Software and Systems. Springer, pp 293–310
Siegl S, Russer M, Hielscher KS (2015) Partitioning the requirements of embedded systems by input/output dependency analysis for compositional creation of parallel test models. In: 2015 Annual IEEE Systems Conference (SysCon) Proceedings, pp 96–102. https://doi.org/10.1109/SYSCON.2015.7116735
da Silveira MB, Rodrigues EdM, Zorzo AF, Costa LT, Vieira HV, de Oliveira FM (2011) Generation of scripts for performance testing based on UML models. In: The 23rd International Conference on Software Engineering and Knowledge Engineering (SEKE), pp 258–263
Smith C, Williams L (1993) Software performance engineering: a case study including performance comparison with design alternatives. IEEE Trans Softw Eng 19(7):720–741. https://doi.org/10.1109/32.238572
Smith CU, Williams LG (2001) Performance solutions: a practical guide to creating responsive, scalable software. Addison-Wesley. Google-Books-ID: X5VlQgAACAAJ
Stol KJ, Fitzgerald B (2018) The ABC of software engineering research. ACM Trans Softw Eng Methodol 27(3):1–51. https://doi.org/10.1145/3241743
Technologies, C.: Classic cases where performance testing failures plagued large organizations. https://www.cigniti.com/blog/2-classic-cases-where-performance-testing-failures-plague-large-organisations/. Accessed 15 Mar 2020
Utting M, Pretschner A, Legeard B (2012) A taxonomy of model-based testing approaches. Softw Test Verif Reliab 22(5):297–312. https://doi.org/10.1002/stvr.456
Utting M, Pretschner A, Legeard B, Utting CM, Pretschner E, Legeard B, Uttinga M, Pretschnerb E, Legeardc B (2006) Legeard b., a taxonomy of model-based testing. Department of Computer Science, The University of Waikato, Hamilton, New Zealand
Vain J, Tsiopoulos L, Kharchenko V, Kaur A, Jenihhin M, Raik J (2017) Multi-fragment Markov model guided online test generation for MPSoC. In: ICTERI 2017 proceedings, p 14
Wang C, Pastore F, Briand L (2017) System testing of timing requirements based on use cases and timed automata. In: 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST), pp 299–309. https://doi.org/10.1109/ICST.2017.34
Wang J, Bai X, Li L, Ji Z, Ma H (2017) A model-based framework for cloud API testing. In: 2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC), vol 2, pp 60–65. https://doi.org/10.1109/COMPSAC.2017.24. ISSN: 0730-3157
Weyns D (2012) Towards an integrated approach for validating qualities of self-adaptive systems. In: Proceedings of the ninth international Workshop on Dynamic Analysis, WODA 2012. Association for Computing Machinery, pp 24–29. https://doi.org/10.1145/2338966.2336803
Wikipedia: Healthcare.gov. https://en.wikipedia.org/wiki/HealthCare.gov. Accessed 15 Mar 2020
Wilke C, Götz S, Reimann J, Aßmann U (2011) Vision paper: towards model-based energy testing. In: Whittle J, Clark T, Kühne T (eds) Model driven engineering languages and systems. Lecture notes in computer science. Springer, Berlin, pp 480–489. https://doi.org/10.1007/978-3-642-24485-8_35
Wohlin C, Runeson, P., Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer. Google-Books-ID: QPVsM1\_U8nkC