A systematic mapping study addressing the reliability of mobile applications: The need to move beyond testing reliability

Tan, 2018, Repairing crashes in android apps, 187 Xia, 2016, Predicting crashing releases of mobile applications, 1 Khalid, 2014, What do mobile app users complain about?, IEEE Softw., 32, 70, 10.1109/MS.2014.50 Kumar, 2013, Mobile health technology evaluation: the mHealth evidence workshop, Am. J. Prev. Med., 45, 228, 10.1016/j.amepre.2013.03.017 Garofalakis, 2007, Quality attributes of consumer-based m-commerce systems, 130 Ghinea, 2004, A user perspective of quality of service in m-commerce, Multimedia Tools Appl., 22, 187, 10.1023/B:MTAP.0000011934.59111.b5 El Zarif, 2020, On the relationship between user churn and software issues, 339 Maiga, 2015, An empirical study on the handling of crash reports in a large software company: An experience report, 342 Salz, 2014, Monitoring mobile app performance, J. Direct Data Digit. Mark. Pract., 15, 219, 10.1057/dddmp.2014.9 Musa, 1990, Software-reliability engineering: technology for the 1990s, IEEE Softw., 7, 36, 10.1109/52.60588 Wang, 2017, An explorative study of the mobile app ecosystem from app developers’ perspective, 163 Su, 2020, Why my app crashes understanding and benchmarking framework-specific exceptions of android apps, IEEE Trans. Softw. Eng. Genc-Nayebi, 2017, A systematic literature review: Opinion mining studies from mobile app store user reviews, J. Syst. Softw., 125, 207, 10.1016/j.jss.2016.11.027 Meskini, 2013, Reliability models applied to mobile applications, 155 Barack, 2020, Assessment and prediction of software reliability in mobile applications, J. Softw. Eng. Appl., 13, 179, 10.4236/jsea.2020.139012 Anand, 2017, Software reliability modeling with impact of beta testing on release decision, 121 Wasserman, 2010, Software engineering issues for mobile application development Fan, 2018, Large-scale analysis of framework-specific exceptions in android apps, 408 Kechagia, 2018, The exception handling riddle: An empirical study on the Android API, J. Syst. Softw., 142, 248, 10.1016/j.jss.2018.04.034 Zein, 2016, A systematic mapping study of mobile application testing techniques, J. Syst. Softw., 117, 334, 10.1016/j.jss.2016.03.065 Riganelli, 2016, Healing data loss problems in android apps, 146 Catolino, 2019, Cross-project just-in-time bug prediction for mobile apps: an empirical assessment, 99 Yang, 2000, A study of operational and testing reliability in software reliability analysis, Reliab. Eng. Syst. Saf., 70, 323, 10.1016/S0951-8320(00)00069-7 Littlewood, 1980, Theories of software reliability: How good are they and how can they be improved?, IEEE Trans. Softw. Eng., 489, 10.1109/TSE.1980.230790 Lyu, 2007, Software reliability engineering: A roadmap, 153 ISO/IEC, 2011 O’connor, 1988, Software reliability: Measurement, prediction, application, J. D. Musa, A. Iannino and K. Okumoto, McGraw-Hill, Qual. Reliab. Eng. Int., 4, 296 Parnas, 2003, The role of inspection in software quality assurance, IEEE Trans. Softw. Eng., 29, 674, 10.1109/TSE.2003.1223642 Febrero, 2014, A systematic mapping study of software reliability modeling, Inf. Softw. Technol., 56, 839, 10.1016/j.infsof.2014.03.006 Vaidyanathan, 2005, A comprehensive model for software rejuvenation, IEEE Trans. Dependable Secure Comput., 2, 124, 10.1109/TDSC.2005.15 Android App Bundles, 2021 App Manifest Overview, 2021 Google, n.d. Android Developer Guide. https://developer.android.com/guide/components/activities/activity-lifecycle (accessed 28.09.2021). U. Farooq, Z. Zhao, 2018. Runtimedroid: Restarting-free runtime change handling for android apps. In: Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services. pp. 110–122. L. Fan, et al. 2018c. Efficiently manifesting asynchronous programming errors in android apps. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. pp. 486–497. Singhal, 2011, A systematic review of software reliability studies, Softw. Eng.: Int. J., 1, 96 Shahrokni, 2013, A systematic review of software robustness, Inf. Softw. Technol., 55, 1, 10.1016/j.infsof.2012.06.002 Alhazzaa, 2019, A systematic mapping study on software reliability growth models that consider evolution, 83 Febrero, 2016, Software reliability modeling based on ISO/IEC SQuaRE, Inf. Softw. Technol., 70, 18, 10.1016/j.infsof.2015.09.006 Ali, 2009, A systematic review of the application and empirical investigation of search-based test case generation, IEEE Trans. Softw. Eng., 36, 742, 10.1109/TSE.2009.52 Kitchenham, 2007 Petersen, 2015, Guidelines for conducting systematic mapping studies in software engineering: An update, Inf. Softw. Technol., 64, 1, 10.1016/j.infsof.2015.03.007 Wimalasooriya, 2021 Maplesden, 2015, Performance analysis for object-oriented software: A systematic mapping, IEEE Trans. Softw. Eng., 41, 691 Cohen, 1960, A coefficient of agreement for nominal scales, Educ. Psychol. Meas., 20, 37, 10.1177/001316446002000104 Landis, 1977, The measurement of observer agreement for categorical data, Biometrics, 33, 159, 10.2307/2529310 Savolainen, 2012, Software development project success and failure from the supplier’s perspective: A systematic literature review, Int. J. Proj. Manage., 30, 458, 10.1016/j.ijproman.2011.07.002 Sahinoglu, 2015, Mobile application verification: a systematic mapping study, 147 Kitchenham, 2010, What’s up with software metrics?–A preliminary mapping study, J. Syst. Softw., 83, 37, 10.1016/j.jss.2009.06.041 Meho, 2007, Impact of data sources on citation counts and rankings of LIS faculty: Web of Science versus Scopus and Google Scholar, J. Am. Soc. Inf. Sci. Technol., 58, 2105, 10.1002/asi.20677 Journals and Wiley Online Library, 2021 Cotroneo, 2019, Dependability assessment of the Android OS through fault injection, IEEE Trans. Reliab. Kumar, 2020, Smartphone processor architecture, operations, and functions: current state-of-the-art and future outlook: energy performance trade-off: Energy–performance trade-off for smartphone processors, J. Supercomput., 1, 10.1007/s11227-018-2503-6 Chen, 2019, An adaptive offloading framework for android applications in mobile edge computing, Sci. China Inf. Sci., 62, 1, 10.1007/s11432-018-9749-8 Liñán, 2018, Automated extraction of augmented models for Android apps, 549 Wang, 2019, Understanding ineffective events and reducing test sequences for android applications, 264 Tramontana, 2019, Automated functional testing of mobile applications: a systematic mapping study, Softw. Qual. J., 27, 149, 10.1007/s11219-018-9418-6 Amalfitano, 2012, A toolset for GUI testing of android applications, 650 Amalfitano, 2011, A gui crawling-based technique for android mobile application testing, 252 Petersen, 2008, Systematic mapping studies in software engineering, 1 Wieringa, 2006, Requirements engineering paper classification and evaluation criteria: a proposal and a discussion, Requir. Eng., 11, 102, 10.1007/s00766-005-0021-6 Wohlin, 2015, Towards a decision-making structure for selecting a research design in empirical software engineering, Empir. Softw. Eng., 20, 1427, 10.1007/s10664-014-9319-7 Sjoberg, 2007, The future of empirical methods in software engineering research, 358 Easterbrook, 2008, Selecting empirical methods for software engineering research, 285 Wohlin, 2003, Empirical research methods in software engineering, 7 Robson, 2002 Wohlin, 2012 Lethbridge, 2005, Studying software engineers: Data collection techniques for software field studies, Empir. Softw. Eng., 10, 311, 10.1007/s10664-005-1290-x Zelkowitz, 1998, Experimental models for validating technology, Computer, 31, 23, 10.1109/2.675630 Runeson, 2009, Guidelines for conducting and reporting case study research in software engineering, Empir. Softw. Eng., 14, 131, 10.1007/s10664-008-9102-8 Kitchenham, 2002, Preliminary guidelines for empirical research in software engineering, IEEE Trans. Softw. Eng., 28, 721, 10.1109/TSE.2002.1027796 Yin, 2003, vol. 5 Shadish, 2002 Runeson, 2012 Garousi, 2016, Challenges and best practices in industry-academia collaborations in software engineering: A systematic literature review, Inf. Softw. Technol., 79, 106, 10.1016/j.infsof.2016.07.006 Al Dallal, 2017, Empirical evaluation of the impact of object-oriented code refactoring on quality attributes: A systematic literature review, IEEE Trans. Softw. Eng., 44, 44, 10.1109/TSE.2017.2658573 Martin, 2016, A survey of app store analysis for software engineering, IEEE Trans. Softw. Eng., 43, 817, 10.1109/TSE.2016.2630689 Stringfellow, 2002, An empirical method for selecting software reliability growth models, Empir. Softw. Eng., 7, 319, 10.1023/A:1020515105175 IEEE Standards Board, 1990 Avizienis, 2001 Ordonez, 2008, The state of metrics in software industry, 453 Ryan, 2005, Software, performance and resource utilisation metrics for context-aware mobile applications, 10pp. Linares-Vásquez, 2017, Continuous, evolutionary and large-scale: A new perspective for automated mobile app testing, 399 Yu, 2018, Localizing function errors in mobile apps with user reviews, 418 Jha, 2017, Developer mistakes in writing android manifests: An empirical study of configuration errors, 25 Hassan, 2008, The road ahead for mining software repositories, 48 dos Santos, 2009, Action research use in software engineering: An initial survey, 414 Scalabrino, 2020, API compatibility issues in Android: Causes and effectiveness of data-driven detection techniques, Empir. Softw. Eng., 25, 5006, 10.1007/s10664-020-09877-w A.S. Ami, M.M. Hasan, M.R. Rahman, K. Sakib, 2018. MobiCoMonkey: Context testing of Android apps. In: Proceedings of the 5th International Conference on Mobile Software Engineering and Systems. pp. 76–79. Paydar, 2017, Experimental study on the importance and effectiveness of monkey testing for android applications, 73 Gómez, 2017, App store 2.0: From crowdsourced information to actionable feedback in mobile ecosystems, IEEE Softw., 34, 81, 10.1109/MS.2017.46 Qiao, 2018, Two-level rejuvenation for android smartphones and its optimization, IEEE Trans. Reliab., 68, 633, 10.1109/TR.2018.2881306 Wu, 2019, Analyses for specific defects in android applications: a survey, Front. Comput. Sci., 1 Almeida, 2019, Testing tools for android context-aware applications: a systematic mapping, J. Braz. Comput. Soc., 25, 1, 10.1186/s13173-019-0093-7 Corral, 2014, Defining relevant software quality characteristics from publishing policies of mobile app stores, 205 Liu, 2014, Software quality testing model for mobile application, 192 Jha, 2019, Mining non-functional requirements from app store reviews, Empir. Softw. Eng., 24, 3659, 10.1007/s10664-019-09716-7 Hassan, 2017, An empirical study of emergency updates for top android mobile apps, Empir. Softw. Eng., 22, 505, 10.1007/s10664-016-9435-7 Zhu, 2019, A formal framework for software faults and permissions based on unified theory of programming, Cluster Comput., 22, 14049, 10.1007/s10586-018-2233-9 Li, 2020, CDA: Characterising deprecated Android APIs, Empir. Softw. Eng., 1 Canfora, 2016, Exploring mobile user experience through code quality metrics, 705 Paydar, 2020, An empirical study on the effectiveness of monkey testing for Android applications, Iran. J. Sci. Technol. Trans. Electr. Eng., 44, 1013, 10.1007/s40998-019-00270-y T. Dey, A. Mockus, 2018. Modeling relationship between post-release faults and usage in mobile software. In: Proceedings of the 14th International Conference on Predictive Models and Data Analytics in Software Engineering. pp. 56–65. L. Li, T.F. Bissyandé, H. Wang, J. Klein, 2018. Cid: Automating the detection of api-related compatibility issues in android apps. In: Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis. pp. 153–163. U. Farooq, Z. Zhao, 2018. Runtimedroid: Restarting-free runtime change handling for android apps. In: Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services. pp. 110–122. De Souza Coelho, 2017, Exception handling bug hazards in android: Results from a mining study and an exploratory survey, Empir. Softw. Eng., 22 Wu, 2017, Measuring the declared SDK versions and their consistency with API calls in Android apps, 678 J. Oliveira, N. Cacho, D. Borges, T. Silva, F. Castor, 2016. An exploratory study of exception handling behavior in evolving android and java applications. In: Proceedings of the 30th Brazilian Symposium on Software Engineering. pp. 23–32. Gomez, 2015, A recommender system of buggy app checkers for app store moderators, 1 A.K. Jha, W.J. Lee, 2013. Capture and replay technique for reproducing crash in android applications. In: Proceedings of the 12th IASTED International Conference in Software Engineering. pp. 783–790. S. Ghari, M. Hadian, M. Rasolroveicy, M. Fokaefs, 2019. A multi-dimensional quality analysis of Android applications. In: Proceedings of the 29th Annual International Conference on Computer Science and Software Engineering. pp. 34–43. Hsiao, 2014, Race detection for event-driven mobile applications, ACM SIGPLAN Not., 49, 326, 10.1145/2666356.2594330 I.T. Mercado, N. Munaiah, A. Meneely, 2016. The impact of cross-platform development approaches for mobile applications from the user’s perspective. In: Proceedings of the International Workshop on App Market Analytics. pp. 43–49. Riganelli, 2019, A benchmark of data loss bugs for android apps, 582 Xiang, 2018, A new software rejuvenation model for android, 293 Qiao, 2018, An empirical study on software aging indicators prediction in Android mobile, 271 Sun, 2019, Automatically capturing and reproducing Android application crashes, 524 Moran, 2016, Automatically discovering, reporting and reproducing android application crashes, 33 J. Harty, M. Müller, 2019. Better android apps using android vitals. In: Proceedings of the 3rd ACM SIGSOFT International Workshop on App Market Analytics. pp. 26–32. Jha, 2019, Characterizing Android-specific crash bugs, 111 Kechagia, 2015, Charting the API minefield using software telemetry data, Empir. Softw. Eng., 20, 1785, 10.1007/s10664-014-9343-7 Zhang, 2016, Crashfuzzer: Detecting input processing related crash bugs in android applications, 1 Huang, 2019, Up-to-crash: Evaluating third-party library updatability on android, 15 Dey, 2020, Deriving a usage-independent software quality metric, Empir. Softw. Eng., 25, 1596, 10.1007/s10664-019-09791-w Jha, 2017, Developer mistakes in writing android manifests: An empirical study of configuration errors, 25 Oliveira, 2018, Do android developers neglect error handling? a maintenance-Centric study on the relationship between android abstractions and uncaught exceptions, J. Syst. Softw., 136, 1, 10.1016/j.jss.2017.10.032 Oliveira, 2018, Droideh: An exception handling mechanism for android applications, 200 Azim, 2019, Dynamic slicing for android, 1154 Grano, 2018, Exploring the integration of user feedback in automated testing of android applications, 72 Shan, 2016, Finding resume and restart errors in android applications, ACM SIGPLAN Not., 51, 864, 10.1145/3022671.2984011 White, 2015, Generating reproducible and replayable bug reports from android application crashes, 48 Riganelli, 2016, Healing data loss problems in android apps, 146 Fan, 2018, Large-scale analysis of framework-specific exceptions in android apps, 408 Luo, 2018, MAD-API: Detection, correction and explanation of API misuses in distributed android applications, 123 P. Kong, L. Li, J. Gao, T.F. Bissyandé, J. Klein, 2019. Mining android crash fixes in the absence of issue-and change-tracking systems. In: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. pp. 78–89. S. Banerjee, L. Clapp, M. Sridharan, 2019. Nullaway: Practical type-based null safety for java. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. pp. 740–750. P. Patel, G. Srinivasan, S. Rahaman, I. Neamtiu, 2018. On the effectiveness of random testing for Android: or how i learned to stop worrying and love the monkey. In: Proceedings of the 13th International Workshop on Automation of Software Test. pp. 34–37. Lenarduzzi, 2018, Prioritizing corrective maintenance activities for android applications: An industrial case study on android crash reports, 133 Zhao, 2019, Recdroid: automatically reproducing android application crashes from bug reports, 128 Tan, 2018, Repairing crashes in android apps, 187 Xiong, 2018, Reproducible interference-aware mobile testing, 36 Gómez, 2016, Reproducing context-sensitive crashes of mobile apps using crowdsourced monitoring, 88 Koroglu, 2018, TCM: test case mutation to improve crash detection in android, 264 Kechagia, 2018, The exception handling riddle: An empirical study on the Android API, J. Syst. Software, 142, 248-270, 10.1016/j.jss.2018.04.034 Bavota, 2014, The impact of api change-and fault-proneness on the user ratings of android apps, IEEE Trans. Softw. Eng., 41, 384, 10.1109/TSE.2014.2367027 M.T. Azim, I. Neamtiu, L.M. Marvel, 2014. Towards self-healing smartphone software via automated patching. In: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering. pp. 623–628. M. Kechagia, D. Spinellis, 2014. Undocumented and unchecked: exceptions that spell trouble. In: Proceedings of the 11th Working Conference on Mining Software Repositories. pp. 312–315. Anbunathan, 2014, A recursive crawler algorithm to detect crash in android application, 1 Nguyen, 2019 O. Riganelli, S.P. Mottadelli, C. Rota, D. Micucci, L. Mariani, 2020. Data loss detector: automatically revealing data loss bugs in Android apps. In: Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. pp. 141–152. Choi, 2015, A lightweight approach to component-level exception mechanism for robust android apps, Comput. Lang. Syst. Struct., 44, 283 Gao, 2015, Fixing recurring crash bugs via analyzing q & a sites (T), 307 M. Cinque, D. Cotroneo, A. Testa, 2012. A logging framework for the on-line failure analysis of android smart phones. In: Proceedings of the 1st European Workshop on AppRoaches to MObiquiTous Resilience. pp. 1–6. Mirzaei, 2015, Exception fault localization in Android applications, 156 L. Fan, T. Su, S. Chen, G. Meng, Y. Liu, L. Xu, G. Pu, 2018a. Efficiently manifesting asynchronous programming errors in android apps. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. pp. 486–497. M. Li, H. Lin, C. Liu, Z. Li, F. Qian, Y. Liu …, T. Xu, 2020. Experience: aging or glitching? why does android stop responding and what can we do about it? In: Proceedings of the 26th Annual International Conference on Mobile Computing and Networking. pp. 1–11. Zhang, 2014, Amplifying tests to validate exception handling code: An extended study in the mobile application domain, ACM Trans. Softw. Eng. Methodol. (TOSEM), 23, 1, 10.1145/2652483 Gu, 2016, Automatic runtime recovery via error handler synthesis, 684 Wu, 2017, Exception beyond exception: Crashing android system by trapping in uncaught exception, 283 Wu, 2021, Scalable online vetting of android apps for measuring declared SDK versions and their consistency with API calls, Empir. Softw. Eng., 26, 1, 10.1007/s10664-020-09897-6 Weng, 2017, A rejuvenation strategy in Android, 273 Weng, 2016, Analysis of software aging in android, 78 Núñez, 2020, A model-driven approach for the development of native mobile applications focusing on the data layer, J. Syst. Softw., 161, 10.1016/j.jss.2019.110489 Liu, 2016, PF-Miner: A practical paired functions mining method for Android kernel in error paths, J. Syst. Softw., 121, 234, 10.1016/j.jss.2016.02.007 X. Xia, E. Shihab, Y. Kamei, D. Lo, X. Wang, 2016a. Predicting crashing releases of mobile applications. In: Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. pp. 1–10. Su, 2020, Why my app crashes understanding and benchmarking framework-specific exceptions of android apps, IEEE Trans. Softw. Eng. Li, 2020, Automated bug reproduction from user reviews for Android applications, 51 L. Pan, B. Cui, H. Liu, J. Yan, S. Wang, J. Yan, J. Zhang, 2020. Static asynchronous component misuse detection for Android applications. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. pp. 952–963. Scalabrino, 2020, API compatibility issues in Android: Causes and effectiveness of data-driven detection techniques, Empir. Softw. Eng., 25, 5006, 10.1007/s10664-020-09877-w Xiang, 2020, Software aging and rejuvenation in android: new models and metrics, Softw. Qual. J., 28, 85, 10.1007/s11219-019-09475-0 Kong, 2021, ANCHOR: locating android framework-specific crashing faults, Autom. Softw. Eng., 28, 1, 10.1007/s10515-021-00290-1 Xie, 2020, Automatically detecting exception handling defects in android applications, 61 He, 2018, Understanding and detecting evolution-induced compatibility issues in Android apps, 167 T. Nguyen, P. Vu, T. Nguyen, 2020. Code recommendation for exception handling. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. pp. 1027–1038. S. Mahajan, N. Abolhassani, M.R. Prasad, 2020. Recommending stack overflow posts for fixing runtime exceptions using failure scenario matching. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. pp. 1052–1064.