Quality evaluation meta-model for open-source software: multi-method validation study
Software Quality Journal - Trang 1-55 - 2024
Tóm tắt
In recent years, open-source software (OSS) has attracted increasing attention due to its easy accessibility via cloud repositories, voluntary community, no vendor lock-in, and low total cost of ownership. In turn, specifying and evaluating OSS quality has become a significant challenge for OSS adoption in organizations that are inclined to use them. Although many OSS quality models have been proposed in the literature, the dynamic and diverse nature of OSS has caused these models to be heterogeneous in terms of structure and content. This has adversely affected the standardization of evaluations and led to the evaluation results obtained from different OSS quality models for the same purpose being incomparable and sometimes unreliable. Therefore, in this study, a meta-model for OSS quality (OSS-QMM), which employs a unified structure from existing quality models and enables the derivation of homogeneous models, has been proposed. For this purpose, a systematic and laborious effort has been spent via a step-based meta-model creation process including review-and-revise iterations. In order to validate the OSS-QMM, case study and expert opinion methods have been applied to answer three research questions (RQs) targeted to investigate practical applicability, results comparability, and effectiveness of using the meta-model. Multiple and embedded case study designs have been employed for evaluating three real ERP systems, and 20 subject matter experts have been interviewed during the validation process. The results of multi-faceted empirical studies have indicated that the OSS-QMM has addressed solving problems in OSS quality evaluation and its adoption with high degrees of confidence.
Tài liệu tham khảo
Adeoye-Olatunde, O. A., & Olenik, N. L. (2021). Research and scholarly methods: Semi-structured interviews. Journal of the American College of Clinical Pharmacy, 4(10), 1358–1367.
Adewumi, A., Misra, S., & Omoragbe, N. (2019). FOSSES: Framework for open- source software evaluation and selection. In: Software: Practice and Experience, 49(5), 780–812.
Adewumi, A., Misra, S., Omoragbe, N., Crawford, B., & Soto, R. (2016). A systematic literature review of open source software quality assessment models. SpringerPlus, 5(1), 1936.
Al-Dhaqm, A., Razak, S., Othman, S. H., Ngadi, A., Ahmed, M. N., & Ali Mohammed, A. (2017). Development and validation of a database forensic metamodel (DBFM). PloS One, 12(2), e0170793.
Alsolai, H., & Roper, M. (2020). A systematic literature review of machine learning techniques for software maintainability prediction. Information and Software Technology, 119, 106214.
Ardito, L., Coppola, R., Barbato, L., & Verga, D. (2020). A tool-based perspective on software code maintainability metrics: A systematic literature review. Scientific Programming, 2020.
Arthur, J. D., & Stevens, K. T. (1989). Assessing the adequacy of documentation through document quality indicators. In Proceedings. Conference on Software Maintenance, 40–49. IEEE.
Aversano, L., & Tortorella, M. (2013). Quality evaluation of floss projects: Application to ERP systems. Information and Software Technology, 55(7), 1260–1276.
Aversano, L., Guardabascio, D., & Tortorella, M. (2017). Analysis of the documentation of ERP software projects. Procedia Computer Science, 121, 423–430.
Bakar, A. D., Sultan, A. B. M., Zulzalil, H., & Din, J. (2012). Review on ‘maintainability’metrics in open source software. International Review on Computers and Software, 7(3), 903–907.
Bayer, J., & Muthig, D. (2006). A view-based approach for improving software documentation practices. 13th Annual IEEE International Symposium and Workshop on Engineering of Computer-Based Systems (ECBS’06) (p. 10). IEEE.
Beydoun, G., Low, G., Henderson-Sellers, B., Mouratidis, H., Gomez-Sanz, J. J., Pavon, J., & Gonzalez-Perez, C. (2009). FAML: A generic metamodel for MAS development. IEEE Transactions on Software Engineering, 35(6), 841–863.
Boehm, B. W., Brown, H., & Lipow, M. (1978). Quantitative evaluation of software quality. In Proceedings of the 2nd International Conference on Software Engineering, 592–605.
Briand, L., Morasca, S., & Basili, V. (2002). An operational process for goal driven definition of measures. IEEE Transactions on Software Engineering, 28(12), 1106–1125.
Brings, J., Daun, M., Keller, K., Obe, P. A., & Weyer, T. (2020). A systematic map on verification and validation of emergent behavior in software engineering research. Future Generation Computer Systems, 112, 1010–1037.
Butler, S., Gamalielsson, J., Lundell, B., Brax, C., Mattsson, A., Gustavsson, T., & Lönroth, E. (2022). Considerations and challenges for the adoption of open source components in software-intensive businesses. Journal of Systems and Software, 186, 111152.
Chakraborty, S. (2022). TOPSIS and modified TOPSIS: A comparative analysis. Decision Analytics Journal, 2, 100021.
Chawla, M. K., & Chhabra, I. (2015, October). Sqmma: Software quality model for maintainability analysis. In Proceedings of the 8th Annual ACM India Conference, 9–17
Chidamber, S. R., & Kemerer, C. F. (1994). A metrics suite for object oriented design. IEEE Transactions on Software Engineering, 20(6), 476–493.
Codemetrics. (2019). URL: https://plugins.jetbrains.com/plugin/12159-codemetrics
Dagpinar, M., & Jahnke, J. H. (2003, November). Predicting maintainability with object-oriented metrics-an empirical comparison. In 10th Working Conference on Reverse Engineering, 2003. WCRE 2003. Proceedings, 155–155. IEEE Computer Society.
Dromey, R. G. (1995). A model for software product quality. IEEE Transactions on Software Engineering, 21(2), 146–162.
Dubey, S. K., & Rana, A. (2011). Assessment of maintainability metrics for object-oriented software system. ACM SIGSOFT Software Engineering Notes, 36(5), 1–7.
Duijnhouwer, F. W., & Widdows, C. (2003). Capgemini expert letter open source maturity model. Retrieved: 30 April 2022. Capgemini. URL: tinyurl.com/yxdbvjk6
Dweiri, F., Kumar, S., Khan, S. A., & Jain, V. (2016). Designing an integrated AHP based decision support system for supplier selection in automotive industry. Expert Systems with Applications, 62, 273–283.
Eghan, E. E., Alqahtani, S. S., Forbes, C., & Rilling, J. (2019). API trustworthiness: An ontological approach for software library adoption. Software Quality Journal, 27(3), 969–1014.
Frantz, R. Z., Rehbein, M. H., Berlezi, R., & Roos-Frantz, F. (2019). Ranking open source application integration frameworks based on maintainability metrics: A review of five-year evolution. Software: Practice and Experience, 49(10), 1531–1549.
Garcia, F., Bertoa, M. F., Calero, C., Vallecillo, A., Ruiz, F., Piattini, M., & Genero, M. (2006). Towards a consistent terminology for software measurement. Information and Software Technology, 48(8), 631–644.
Gezici, B., Özdemir, N., Yılmaz, N., Coşkun, E., Tarhan, A., & Chouseinoglou, O. (2019). Quality and success in open source software: A systematic mapping. 45th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), 363–370. IEEE.
Goeb, A. (2013). A meta model for software architecture conformance and quality assessment. Electronic Communications of the EASST, 60.
Grady, R. B. (1992). Practical software metrics for project management and process improvement. Prentice Hall.
Hanefi Calp, M., Arici, N., Enstitüsü, B., Üniversitesi, G., & Ankara, T. (2011). Nesne Yönelimli Tasarım Metrikleri ve Kalite Özellikleriyle İlişkisi. Politeknik Dergisi Journal of Polytechnic Cilt Digital Object Identifier, 14141(10), 9–14.
Hanine, M., Boutkhoum, O., Tikniouine, A., & Agouti, T. (2016). Application of an integrated multi-criteria decision making AHP-TOPSIS methodology for ETL software selection. Springerplus, 5(1), 1–17.
Hasnain, S., Ali, M. K., Akhter, J., Ahmed, B., & Abbas, N. (2020). Selection of an industrial boiler for a soda-ash production plant using analytical hierarchy process and TOPSIS approaches. Case Studies in Thermal Engineering, 19, 100636.
Hauge, O., Osterlie, T., & Sorensen, C. F. (2009) An empirical study on selection of open source software-preliminary results. In: ICSE Workshop on Emerging Trends in Free/Libre/Open Source Software Research and Development. IEEE.
Hmood, A., Keivanloo, I., & Rilling, J. (2012, July). SE-EQUAM-an evolvable quality metamodel. In 2012 IEEE 36th Annual Computer Software and Applications Conference Workshops, 334–339. IEEE
Ho, W., & Ma, X. (2018). The state-of-the-art integrations and applications of the analytic hierarchy process. European Journal of Operational Research, 267(2), 399–414.
IEEE standard glossary of software engineering terminology. (1990). IEEE Standart 610.12-1990. pp. 1–84.
IEEE Standard for a Software Quality Metrics Methodology. (1998). In IEEE Standart 1061–1998.
Işıklar, G., & Büyüközkan, G. (2007). Using a multi-criteria decision making approach to evaluate mobile phone alternatives. Computer Standards & Interfaces, 29(2), 265–274.
ISO, International Standard ISO VIM. (1993). International vocabulary of basic and general terms in metrology, International Standards Organization, Geneva, Switzerland, second edition.
ISO/IEC 14598-3:1999. (1999). Information technology-software product evaluation-Part 3: Process for developers. International Organization for Standardization, Geneva.
ISO/IEC 15939:2007. (2007). Information Technology—Software Engineering—Software Measurement Process. International Organization for Standardization, Geneva.
ISO/IEC 9126-1:2001. (2001). Software engineering - Product quality -Part 1: Quality model, international organization for standardization, Geneva, Switzerland.
Jha, S., Kumar, R., Abdel-Basset, M., Priyadarshini, I., Sharma, R., & Long, H. V. (2019). Deep learning approach for software maintainability metrics prediction. Ieee Access, 7, 61840–61855.
Jiang, S., Cao, J., & Qi, Q. (2021). Exploring development-related factors affecting the popularity of open source software projects. In 2021 IEEE 24th International Conference on Computer Supported Cooperative Work in Design (CSCWD), 244–249. IEEE.
Joshi, A., Kale, S., Chandel, S., & Pal, D. K. (2015). Likert scale: Explored and explained. British Journal of Applied Science & Technology, 7(4), 396.
Khashei-Siuki, A., & Sharifan, H. (2020). Comparison of AHP and FAHP methods in determining suitable areas for drinking water harvesting in Birjand aquifer. Iran. Groundwater for Sustainable Development, 10, 100328.
Khatri, S. K., & Singh, I. (2016). Evaluation of open source software and improving its quality. 5th International Conference on Reliability, Infocom Technologies and Optimization (ICRITO). IEEE.
Kim, H. M. (1999). Representing and reasoning about quality using enterprise models. PhD thesis, Dept. Mechanical and Industrial Engineering, University of Toronto, Canada.
Kitchenham, B., Hughes, R. T., & Linkman, S. G. (2001). Modeling software measurement data. IEEE Transactions on Software Engineering, 27(9), 788–804.
Kläs, M., Lampasona, C., Nunnenmacher, S., Wagner, S., Herrmannsdörfer, M., & Lochmann, K. (2010). How to evaluate meta-models for software quality. In Proceedings of the 20th International Workshop on Software Measurement.
Lenarduzzi, V., Taibi, D., Tosi, D., Lavazza, L., & Morasca, S. (2020). Open source software evaluation, selection, and adoption: A systematic literature review. In: 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE
Li, J., Conradi, R., Bunse, C., Torchiano, M., Slyngstad, O. P. N., & Morisio, M. (2009). Development with off-the-shelf components: 10 facts. IEEE Software, 26(2), 80–87.
List of questions and expert opinion (Step-5). (2023). Retrieved: 20 June 2023. URL: https://tinyurl.com/2ow7ayua
Magaldi, D., & Berler, M. (2020). Semi-structured interviews. Encyclopedia of Personality and Individual Differences, 4825–4830.
Mc Call, J. A., Richards, P. K., & Walters, G. F. (1977). Factors in software quality, volumes I, II, and III. US Rome Air Development Center Reports, US Department of Commerce, USA.
Mens, T., Doctors, L., Habra, N., Vanderose, B., & Kamseu, F. (2011). Qualgen: Modeling and analysing the quality of evolving software systems. In: 15th European Conference on Software Maintenance and Reengineering. IEEE.
MetricsReloaded. (2004). URL: https://plugins.jetbrains.com/plugin/93-metricsreloaded
Nistala, P., Nori, K. V., & Reddy, R. (2019). Software quality models: A systematic mapping study. International Conference on Software and System Processes (ICSSP), 125–134. IEEE.
Object Management Group (OMG). (2019). Meta Object Facility (MOF). Core specification version 2.5.1. Retrieved: 2 October 2022. URL: https://www.omg.org/spec/MOF/2.5.1/PDF
Othman, S. H., & Beydoun, G. (2010). Metamodelling approach to support disaster management knowledge sharing. In: 21st Australasian Conference on Information Systems.
Othman, S. H., Beydoun, G., & Sugumaran, V. (2014). Development and validation of a disaster management metamodel (DMM). Information Processing & Management, 50(2), 235–271.
Samoladas, I., Goussios, G., & Spinellis, D. (2008). The SQO-OSS quality model: measurement based open source software evaluation. In: IFIP International Conference on Open Source Systems. Springer, Boston, MA.
Saaty, T. L. (1980). The analytic hierarchy process: Planning, priority setting and resource allocation. McGraw-Hill.
Saaty, T. L. (2008). Decision making with the analytic hierarchy process. International Journal of Services Sciences, 1(1), 83–98.
Saaty, T. L., & Sagir, M. (2015). Ranking countries more reliably in the summer olympics. International Journal of the Analytic Hierarchy Process, 7(3), 589–610.
Salem, I. E. B. (2015). Transformational leadership: Relationship to job stress and job burnout in five-star hotels. Tourism and Hospitality Research, 15(4), 240–253.
SciTools Understand. (2020). URL. https://scitools.com/
Semeteys, R. (2006). Method for qualification and selection of open source software (QSOS), version 1.6. Retrieved: 30 April 2022. URL: tinyurl.com/y2phllex
Silva, D. G., Coutinho, C., & Costa, C. J. (2023). Factors influencing free and open-source software adoption in developing countries—an empirical study. Journal of Open Innovation: Technology, Market, and Complexity, 9(1), 21–33.
Sjoberg, G., Orum, A. M., & Feagin, J. R. (2020). A case for the case study. The University of North Carolina Press.
Soto, M., & Ciolkowski, M. (2009). The QualOSS open source assessment model measuring the performance of open source communities. In: 3rd International Symposium on Empirical Software Engineering and Measurement. IEEE.
Spinellis, D., & Jureczko, M. (May 2011). Metric Description [Online] Available: http://gromit.iiar.pwr.wroc.pl/p_inf/ckjm/
Swedberg, R. (2020). Exploratory research. The Production of Knowledge: Enhancing Progress in Social Science, 17–41.
Tanrıöver, Ö. Ö., & Bilgen, S. (2011). A framework for reviewing domain specific conceptual models. Computer Standards & Interfaces, 33(5), 448–464.
Tassone, J., Xu, S., Wang, C., Chen, J., & Du, W. (2018) Quality assessment of open source software: A review. IEEE/ACIS 17th International Conference on Computer and Information Science (ICIS), 411–416. IEEE.
Vanderose, B., Habra, N., & Kamseu, F. (2010). Towards a model-centric quality assessment. In Proceedings of the 20th International Workshop on Software Measurement (IWSM 2010): Conference on Software Process and Product Measurement (Stuttgart Nov 2010).
Visconti, M., & Cook, C. R. (2002) An overview of industrial software documentation practice. In 12th International Conference of the Chilean Computer Science Society, 2002. Proceedings, 179–186. IEEE.
Wagner, S., Goeb, A., Heinemann, L., Kläs, M., Lampasona, C., Lochmann, K., Mayr, A., Plösch, R., Seidl, A., Streit, J., & Trendowicz, A. (2015). Operationalised product quality models and assessment: The Quamoco approach. Information and Software Technology, 62, 101–123.
Wasserman, M. P., & Chan, C. (2006). Business readiness rating project, BRR Whitepaper RFC 1. URL: tinyurl.com/y5srd5sq
Wasserman, A. I., Guo, X., McMillian, B., Qian, K., Wei, M. Y., & Xu, Q. (2017). OSSpal: Finding and evaluating open source software. In Open Source Systems: Towards Robust Practices: 13th IFIP WG 2.13 International Conference.
Wohlin, C. (2021). Case study research in software engineering—it is a case, and it is a study, but is it a case study? Information and Software Technology, 133, 106514.
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2012). Experimentation in software engineering: An introduction. Springer.
Yalcin, A. S., Kilic, H. S., & Delen, D. (2022). The use of multi-criteria decision-making methods in business analytics: A comprehensive literature review. Technological Forecasting and Social Change, 174, 121193.
Yin, R. K. (2018). The case study research and applications. Sage.
Yılmaz, N., & Tarhan, A. K. (2020). Meta-models for software quality and its evaluation: A systematic literature review. In: International Workshop on Software Measurement and the 15th International Conference on Software Process and Product Measurement, Mexico.
Yılmaz, N., & Tarhan, A. K. (2022a). Quality evaluation models or frameworks for open source software: A systematic literature review. Journal of Software: Evolution and Process, 34(6), e2458. https://doi.org/10.1002/smr.2458
Yılmaz, N., & Tarhan, A. K. (2022b). Matching terms of quality models and meta-models: Toward a unified meta-model of OSS quality. Software Quality Journal, 1–53. https://doi.org/10.1007/s11219-022-09603-3
Yilmaz, N., & Tarhan, A. K. (2022c). Definition of the term used in the SQMM. Zenodo. https://doi.org/10.5281/zenodo.6367596
Yilmaz, N., & Tarhan, A. K. (2023). Supplementary document of the article titled ‘quality evaluation meta-model for open source software. Zenodo. https://doi.org/10.5281/zenodo.7986369
Zhao, Y., Liang, R., Chen, X., & Zou, J. (2021). Evaluation indicators for open-source software: A review. Cybersecurity, 4(1), 1–24.