Log4Perf: suggesting and updating logging locations for web-based systems’ performance monitoring
Tóm tắt
Performance assurance activities are an essential step in the release cycle of software systems. Logs have become one of the most important sources of information that is used to monitor, understand and improve software performance. However, developers often face the challenge of making logging decisions, i.e., neither logging too little and logging too much is desirable. Although prior research has proposed techniques to assist in logging decisions, those automated logging guidance techniques are rather general, without considering a particular goal, such as monitoring software performance. In this paper, we present Log4Perf, an automated approach that provides suggestions of where to insert logging statement with the goal of monitoring web-based systems’ CPU usage. In the first part of our approach, we leverage the performance model’s prediction errors to suggest the need for updating logging locations when software evolves. In the second part of our approach, we build and manipulate a statistical performance model to identify the locations in the source code that statistically significantly influence CPU usage. To evaluate Log4Perf, we conduct case studies on two open source systems, i.e., CloudStore and OpenMRS, and one large-scale commercial system. Our evaluation results show that our approach can suggest the need for updating logging locations and identify the logging locations that can be kept unchanged. We manually examine the logging locations that are newly suggested or deprecated. We find that our approach can build well-fit statistical performance models, indicating that such models can be leveraged to investigate the influence of locations in the source code on performance. The suggested logging locations are often in small and simple methods that do not have logging statements, and are not performance hotspots. Our approach can be used to complement traditional approaches that are based on software metrics or performance hotspots. In addition, we identify seven root-causes of these suggested or deprecated logging locations. Log4Perf is integrated into the release engineering process of the commercial software to provide logging suggestions on a regular basis.
Tài liệu tham khảo
The Java HotSpot Performance Engine Architecture (2010) Slowly Changing Dimensions. https://www.oracle.com/technetwork/java/whitepaper-135217.html. Oracle
.NET Compiler Platform (”Roslyn”) (2017) .NET Compiler Platform (”Roslyn”). https://github.com/dotnet/roslyn
Ahmed TM, Bezemer C, Chen TH, Hassan AE, Shang W (2016) Studying the effectiveness of application performance management (apm) tools for detecting performance regressions for web applications: an experience report. In: Proceedings of the 13th International Conference on Mining Software Repositories, MSR ’16. ACM, New York, pp 1–12
Alghmadi HM, Syer MD, Shang W, Hassan AE (2016) An automated approach for recommending when to stop performance tests. In: 2016 IEEE International conference on software maintenance and evolution (ICSME), pp 279–289
Apache JMeter (2015) Apache: Jmeter. http://jmeter.apache.org/. Accessed: 2015-06-01
Bezemer C, Milon E, Zaidman A, Pouwelse J (2014) Detecting and analyzing i/o performance regressions. Journal of Software: Evolution and Process 26 (12):1193–1212. https://doi.org/10.1002/smr.1657. https://onlinelibrary.wiley.com/doi/abs/10.1002/smr.1657
Bezemer CP, Zaidman A (2014) Performance optimization of deployed software-as-a-service applications. J Syst Softw 87:87–103. https://doi.org/10.1016/j.jss.2013.09.013
Bootstrap (2017) Bootstrap. https://cran.r-project.org/web/packages/bootstrap/bootstrap.pdf. Accessed: 2017-02-27
Brebner PC (2016) Automatic performance modelling from application performance management (apm) data: an experience report. In: Proceedings of the 7th ACM/SPEC on International Conference on Performance Engineering, ICPE ’16. ACM, New York, pp 55–61
Chen J, Shang W (2017) An exploratory study of performance regression introducing code changes. In: 2017 IEEE international conference on software maintenance and evolution (ICSME). IEEE, pp 341–352
Chen TH, Shang W, Hassan AE, Nasser M, Flora P (2016) Cacheoptimizer: Helping developers configure caching frameworks for hibernate-based database-centric web applications. In: Proceedings of the 24th ACM SIGSOFT international symposium on the foundations of software engineering, FSE ’16
Chen TH, Shang W, Hassan AE, Nasser M, Flora P (2016) Cacheoptimizer: Helping developers configure caching frameworks for hibernate-based database-centric web applications. In: Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering, FSE 2016. ACM, New York, pp 666–677
Chen TH, Shang W, Jiang ZM, Hassan AE, Nasser M, Flora P (2014) Detecting performance anti-patterns for applications developed using object-relational mapping. In: Proceedings of the 36th international conference on software engineering, ICSE 2014. ACM, New York, pp 1001–1012
Chen TH, Shang W, Jiang ZM, Hassan AE, Nasser M, Flora P (2016) Finding and evaluating the performance impact of redundant data access for applications that are developed using object-relational mapping frameworks. IEEE Trans Softw Eng PP(99):1–1
Cliff N (1993) Dominance statistics: ordinal analyses to answer ordinal questions. Psychol Bull 114(3):494
CloudScale (2017) CloudScale Project. http://www.cloudscale-project.eu/
Cohen I, Chase JS, Goldszmidt M, Kelly T, Symons J (2004) Correlating instrumentation data to system states: a building block for automated diagnosis and control. In: OSDI, vol 4, pp 16–16
Cohen I, Zhang S, Goldszmidt M, Symons J, Kelly T, Fox A (2005) Capturing, indexing, clustering, and retrieving system history. In: Proceedings of the twentieth ACM symposium on operating systems principles, SOSP ’05. ACM, New York, pp 105–118
Eclipse JDT (2017) Eclipse Java development tools (JDT). http://www.eclipse.org/jdt/
Farshchi M, Schneider JG, Weber I, Grundy J (2015) Experience report: Anomaly detection of cloud application operations using log and cloud metric correlation analysis. In: 2015 IEEE 26th international symposium on software reliability engineering (ISSRE), pp 24–34
Foo KC, Jiang ZM, Adams B, Hassan AE, Zou Y, Flora P (2010) Mining performance regression testing repositories for automated performance analysis. In: 2010 10th international conference on quality software (QSIC). IEEE, pp 32–41
Freedman DA (2009) Statistical models: theory and practice. Cambridge University Press, Cambridge
Fu Q, Zhu J, Hu W, Lou JG, Ding R, Lin Q, Zhang D, Xie T (2014) Where do developers log? an empirical study on logging practices in industry. In: Companion proceedings of the 36th international conference on software engineering, ICSE companion 2014. ACM, New York, pp 24–33
Gao R, Jiang ZM, Barna C, Litoiu M (2016) A framework to evaluate the effectiveness of different load testing analysis techniques. In: 2016 IEEE International conference on software testing, verification and validation (ICST), pp 22–32
Harrell F (2001) Regression modeling strategies 2001. Springer CrossRef Google Scholar, Nashville
Van Hoorn A, Rohr M, Hasselbring W, Waller J, Ehlers J, Frey S, Kieselhorst D. (2009) Continuous monitoring of software services: Design and application of the Kieker framework
Van Hoorn A, Waller J, Hasselbring W (2012) Kieker: a framework for application performance monitoring and dynamic software analysis. In: Proceedings of the 3rd ACM/SPEC international conference on performance engineering, ICPE ’12. ACM, New York, pp 247–248
Jiang ZM, Hassan AE, Hamann G, Flora P (2009) Automated performance analysis of load tests. In: ICSM ’09: 25th IEEE international conference on software maintenance
JIT (2018) Tracing just-in-time compilation. https://en.wikipedia.org/wiki/Tracing_just-in-time_compilation
JProfiler (2017) JProfiler. https://www.ej-technologies.com/products/jprofiler/overview.html
JVM (2018) Jvm jit compilation as a way of performance optimisation. https://jakubstransky.com/2018/01/15/java-jvm-jit-compilation-performance-optimisation/
Kabinna S, Bezemer C, Shang W, Hassan AE (2016) Logging library migrations: a case study for the apache software foundation projects. In: Proceedings of the 13th international conference on mining software repositories. ACM, pp 154–164
Kabinna S, Bezemer C, Shang W, Syer MD, Hassan AE (2018) Examining the stability of logging statements. Empir Softw Eng 23(1):290–333
Kabinna S, Shang W, Bezemer C, Hassan AE (2016) Examining the stability of logging statements. In: 2016 IEEE 23rd international conference on software analysis, evolution, and reengineering (SANER), vol 1, pp 326–337, DOI https://doi.org/10.1109/SANER.2016.29
Kernighan BW, Pike R (1999) The practice of programming. Addison-Wesley Longman Publishing Co., Inc., Boston
Kuhn M (2008) Building predictive models in r using the caret package. Journal of Statistical Software Articles 28(5):1–26
Li H, Chen THP, Hassan AE, Nasser M, Flora P (2018) Adopting autonomic computing capabilities in existing large-scale systems: an industrial experience report. In: Proceedings of the 40th international conference on software engineering: software engineering in practice. ACM, pp 1–10
Li H, Shang W, Hassan AE (2017) Which log level should developers choose for a new logging statement? Empir Softw Eng 22(4):1684–1716
Li H, Shang W, Zou Y, Hassan AE (2017) Towards just-in-time suggestions for log changes. Empir Softw Eng 22(4):1831–1865
Log4J Async (2017) Log4J Async. https://logging.apache.org/log4j/2.x/manual/async.html
Maplesden D, von Randow K, Tempero E, Hosking J, Grundy J (2015) Performance analysis using subsuming methods: an industrial case study. In: Proceedings of the 37th international conference on software engineering - Volume 2, ICSE ’15. IEEE Press, Piscataway, pp 149–158
Maplesden D, Tempero E, Hosking J, Grundy JC (2015) Subsuming methods: Finding new optimisation opportunities in object-oriented software. In: Proceedings of the 6th ACM/SPEC international conference on performance engineering, ICPE ’15. ACM, New York, pp 175–186
Mockus A (2010) Organizational volatility and its effects on software defects. In: Proceedings of the Eighteenth ACM SIGSOFT international symposium on foundations of software engineering, FSE ’10. ACM, New York, pp 117–126
Moore DS, Craig BA, McCabe GP (2012) Introduction to the practice of statistics. WH Freeman, New York
Nguyen TH, Adams B, Jiang ZM, Hassan AE, Nasser M, Flora P (2012) Automated detection of performance regressions using statistical process control techniques. In: Proceedings of the 3rd ACM/SPEC international conference on performance engineering, ICPE ’12. ACM, New York, pp 299–310
PSUtil (2017) PSUtil. https://github.com/giampaolo/psutil
Sandoval Alcocer JP, Bergel A, Valente MT (2016) Learning from source code history to identify performance failures. In: Proceedings of the 7th ACM/SPEC on international conference on performance engineering, ICPE ’16. ACM, New York, pp 37–48, DOI https://doi.org/10.1145/2851553.2851571
Shang W, Hassan AE, Nasser M, Flora P (2015) Automated detection of performance regressions using regression models on clustered performance counters. In: Proceedings of the 6th ACM/SPEC international conference on performance engineering, ICPE ’15. ACM, New York, pp 15–26
Shang W, Jiang ZM, Adams B, Hassan AE, Godfrey MW, Nasser M, Flora P (2011) An exploratory study of the evolution of communicated information about the execution of large software systems. In: 2011 18Th working conference on reverse engineering, pp 335–344. https://doi.org/10.1109/WCRE.2011.48
Shang W, Jiang ZM, Adams B, Hassan AE, Godfrey MW, Nasser M, Flora P (2014) An exploratory study of the evolution of communicated information about the execution of large software systems. Journal of Software: Evolution and Process 26 (1):3–26
Shihab E, Jiang ZM, Ibrahim WM, Adams B, Hassan AE (2010) Understanding the impact of code and process metrics on post-release defects: a case study on the eclipse project. In: Proceedings of the 2010 ACM-IEEE international symposium on empirical software engineering and measurement, ESEM ’10. ACM, New York, pp 4:1–4:10
Shihab E, Mockus A, Kamei Y, Adams B, Hassan AE (2011) High-impact defects: a study of breakage and surprise defects. In: Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on foundations of software engineering, ESEC/FSE ’11. ACM, New York, pp 300–310
Sim SE, Easterbrook S, Holt RC (2003) Using benchmarking to advance research: a challenge to software engineering. In: proceedings of the 25th international conference on software engineering. IEEE Computer Society, pp 74–83
Stewart C, Kelly T, Zhang A (2007) Exploiting nonstationarity for performance prediction. In: ACM SIGOPS operating systems review, vol 41. ACM, pp 31–44
Syer MD, Shang W, Jiang ZM, Hassan AE (2017) Continuous validation of performance test workloads. Autom Softw Eng 24(1):189–231. https://doi.org/10.1007/s10515-016-0196-8
TPC-W (2017) TPC Benchmark W (TPC-W). http://www.tpc.org/tpcw/
Visual Studio Profiling (2017) Visual Studio Profiling. https://docs.microsoft.com/en-us/visualstudio/profiling
Waller J, Ehmke NC, Hasselbring W (2015) Including performance benchmarks into continuous integration to enable devops. ACM SIGSOFT Software Engineering Notes 40(2):1–4
Weyuker E, Vokolos F (2000) Experience with performance testing of software systems: issues, an approach, and case study. Transactions on Software Engineering 26 (12):1147–1156
Xiong P, Pu C, Zhu X, Griffith R (2013) vperfguard: an automated model-driven framework for application performance diagnosis in consolidated cloud environments. In: Proceedings of the 4th ACM/SPEC international conference on performance engineering, ICPE ’13. ACM, New York, pp 271–282
Yao K, B de Pádua G, Shang W, Sporea S, Toma A, Sajedi S (2018) Log4perf: Suggesting logging locations for web-based systems’ performance monitoring. In: Proceedings of the 2018 ACM/SPEC international conference on performance engineering. ACM, pp 127–138
Yin F, Dong D, Li S, Guo J, Chow K (2018) Java performance troubleshooting and optimization at Alibaba. In: Proceedings of the 40th international conference on software engineering: software engineering in practice, ICSE-SEIP ’18. ACM, New York, pp 11–12, DOI https://doi.org/10.1145/3183519.3183536
Yuan D, Luo Y, Zhuang X, Rodrigues GR, Zhao X, Zhang Y, Jain PU, Stumm M (2014) Simple testing can prevent most critical failures: an analysis of production failures in distributed data-intensive systems. In: Proceedings of the 11th USENIX conference on operating systems design and implementation, OSDI’14. USENIX Association, Berkeley, pp 249–265
Yuan D, Park S, Huang P, Liu Y, Lee MM, Tang X, Zhou Y, Savage S (2012) Be conservative: Enhancing failure diagnosis with proactive logging. In: OSDI ’12: Proceedings of the 10th USENIX conference on operating systems design and implementation, vol 12, pp 293–306
Yuan D, Zheng J, Park S, Zhou Y, Savage S (2011) Improving software diagnosability via log enhancement. In: ASPLOS ’11: Proceedings of the 16th international conference on architectural support for programming languages and operating systems
Zhang S, Ernst MD (2014) Which configuration option should i change?. In: Proceedings of the 36th international conference on software engineering, ICSE 2014. ACM, New York, pp 152–163
Zhang S, Ernst MD (2015) Proactive detection of inadequate diagnostic messages for software configuration errors. In: Proceedings of the 2015 international symposium on software testing and analysis, ISSTA 2015. ACM, New York, pp 12–23
Zhao X, Rodrigues K, Luo Y, Stumm M, Yuan D, Zhou Y (2017) The game of twenty questions: Do you know where to log?. In: Proceedings of the 16th workshop on hot topics in operating systems, HotOS ’17. ACM, New York, pp 125–131
Zhu J, He P, Fu Q, Zhang H, Lyu MR, Zhang D (2015) Learning to log: Helping developers make informed logging decisions. In: Proceedings of the 37th international conference on software engineering - volume 1, ICSE ’15. IEEE Press, Piscataway, pp 415–425
Zhuang Z, Ramachandra H, Tran C, Subramaniam S, Botev C, Xiong C, Sridharan B (2015) Capacity planning and headroom analysis for taming database replication latency: Experiences with linkedin internet traffic. In: Proceedings of the 6th ACM/SPEC international conference on performance engineering, ICPE ’15. ACM, New York, pp 39–50