On negative results when using sentiment analysis tools for software engineering research

Abbasi A, Hassan A, Dhar M (2014) Benchmarking Twitter sentiment analysis tools. In: International Conference on Language Resources and Evaluation. ELRA, Reykjavik, Iceland, pp 823–829 Agarwal A, Xie B, Vovsha I, Rambow O, Passonneau R (2011) Sentiment Analysis of Twitter Data. In: Proceedings of the Workshop on Languages in Social Media, LSM ’11, pp 30–38. Association for Computational Linguistics, Stroudsburg, PA, USA. http://dl.acm.org/citation.cfm?id=2021109.2021114 Asaduzzaman M, Bullock MC, Roy CK, Schneider KA Bug introducing changes: A case study with android. In: Lanza et al. [43], pp 116–119. doi:10.1109/MSR.2012.6224267 Baccianella S, Esuli A, Sebastiani F (2010) SentiWordNet 3.0: an enhanced lexical resource for sentiment analysis and opinion mining. In: Proceedings of the Seventh Conference on International Language Resources and Evaluation (LREC’10). European Language Resources Association (ELRA), Valletta, Malta. http://www.lrec-conf.org/proceedings/lrec2010/pdf/769_Paper.pdf Bakeman R, Gottman JM (1997) Observing interaction: an introduction to sequential analysis. Cambridge University Press. https://books.google.nl/books?id=CMj2SmcijhEC Barkmann H, Lincke R, Löwe W (2009) Quantitative evaluation of software quality metrics in open-source projects. In: IEEE International Workshop on Quantitative Evaluation of large-scale Systems and Technologies, pp 1067–1072 Batista GEAPA, Carvalho ACPLF, Monard MC (2000) Applying one-sided selection to unbalanced datasets. In: Mexican International Conference on Artificial Intelligence: Advances in Artificial Intelligence. Springer-Verlag, London, UK, UK, pp 315–325 Beck K, Beedle M, van Bennekum A, Cockburn A, Cunningham W, Fowler M, Grenning J, Highsmith J, Hunt A, Jeffries R, Kern J, Marick B, Martin RC, Mellor S, Schwaber K, Sutherland J, Thomas D (2001) Manifesto for agile software development http://agilemanifesto.org/principles.html Last accessed: October 14, 2015 Bird S, Loper E, Klein E (2009) Natural language processing with Python. O’Reilly Media Inc Brunner E, Munzel U (2000) The Nonparametric Behrens-Fisher Problem: Asymptotic Theory and a Small-Sample Approximation. Biometrical Journal 42(1):17–25 Capiluppi A, Serebrenik A, Singer L (2013) Assessing technical candidates on the social web. Software. IEEE 30(1):45–51. doi:10.1109/MS.2012.169 Cataldo M, Herbsleb JD (2008) Communication networks in geographically distributed software development. In: Proceedings of the 2008 ACM conference on Computer supported cooperative work, CSCW ’08, pp 579–588. ACM, New York, NY, USA. doi:10.1145/1460563.1460654 Cohen J (1968) Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull 70(4):213–220 Costa JM, Cataldo M, de Souza CR (2011) The scale and evolution of coordination needs in large-scale distributed projects: implications for the future generation of collaborative tools. In: Proceedings of the 2011 annual conference on Human factors in computing systems, CHI ’11, pp 3151–3160. ACM, New York, NY, USA. doi:10.1145/1978942.1979409 Dajsuren Y, van den Brand MGJ, Serebrenik A, Roubtsov S (2013) Simulink models are also software: Modularity assessment. In: Proceedings of the 9th International ACM Sigsoft Conference on Quality of Software Architectures, QoSA ’13, pp 99–106. ACM, New York, NY, USA. doi:10.1145/2465478.2465482 Danescu-Niculescu-Mizil C, Sudhof M, Jurafsky D, Leskovec J, Potts C (2013) A computational approach to politeness with application to social factors. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, 4-9 August 2013, Sofia, Bulgaria, Volume 1: Long Papers, pp 250–259. The Association for Computer Linguistics. http://aclweb.org/anthology/P/P13/P13-1025.pdf Datta S, Sindhgatta R, Sengupta B (2012) Talk versus work: characteristics of developer collaboration on the Jazz platform. In: Proceedings of the ACM international conference on Object oriented programming systems languages and applications, OOPSLA ’12, pp 655–668. ACM, New York, NY, USA. doi:10.1145/2384616.2384664 Davidov D, Tsur O, Rappoport A (2010) Semi-supervised recognition of sarcastic sentences in Twitter and Amazon. In: Proceedings of the Fourteenth Conference on Computational Natural Language Learning, CoNLL ’10, pp. 107–116. Association for Computational Linguistics, Stroudsburg, PA, USA. http://dl.acm.org/citation.cfm?id=1870568.1870582 de Magalhães CVC, da Silva FQB, Santos RES (2014) Investigations about replication of empirical studies in software engineering: Preliminary findings from a mapping study. In: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, EASE ’14, pp 37:1–37:10. ACM, New York, NY, USA. doi:10.1145/2601248.2601289 Destefanis G, Ortu M, Counsell S, Swift S, Marchesi M, Tonelli R (2016). Peer J Comput Sci 2(e73):1–35. doi:10.7717/peerj-cs.73 Dewan P (2015) Towards Emotion-Based Collaborative Software Engineering. In: 2015 IEEE/ACM 8th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE), pp 109–112. doi:10.1109/CHASE.2015.32 Dunn OJ (1961) Multiple comparisons among means. J Am Stat Assoc 56(293):52–64 Dyer R, Nguyen HA, Rajan H, Nguyen TN (2013) Boa: A language and infrastructure for analyzing ultra-large-scale software repositories. In: Proceedings of the 2013 International Conference on Software Engineering, ICSE ’13, pp 422–431. IEEE Press, Piscataway, NJ, USA. http://dl.acm.org/citation.cfm?id=2486788.2486844 Fleiss JL, Levin B, Paik MC (2003) Statistical methods for rates and proportions, 3rd edn. Wiley Series in Probability and Statistics. Wiley, Hoboken, NJ Fontana FA, Mariani E, Morniroli A, Sormani R, Tonello A (2011) An experience report on using code smells detection tools. In: ICST Workshops, pp 450–457. IEEE Fuhr N, Muller P (1987) Probabilistic search term weighting - some negative results. In: Proceedings of the 10th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’87, pp 13–18. ACM, New York, NY, USA. doi:10.1145/42005.42007 Gabriel KR (1969) Simultaneous test procedures—some theory of multiple comparisons. Ann Math Stat 40(1):224–250 Gamon M, Aue A, Corston-Oliver S, Ringger E (2005) Pulse: Mining customer opinions from free text. In: Proceedings of the 6th International Conference on Advances in Intelligent Data Analysis, IDA’05. Springer-Verlag, Berlin, Heidelberg, pp 121–132. doi:10.1007/11552253_12 Garcia D, Zanetti MS, Schweitzer F (2013) The role of emotions in contributors activity: A case study on the Gentoo community. In: International Conference on Cloud and Green Computing, pp 410–417 Giraud-Carrier C, Dunham MH (2011) On the importance of sharing negative results. Sigkdd Explor. Newsl. 12(2):3–4. doi:10.1145/1964897.1964899 Goul M, Marjanovic O, Baxley S, Vizecky K (2012) Managing the Enterprise Business Intelligence App Store: Sentiment Analysis Supported Requirements Engineering. In: 2012 45th Hawaii International Conference on System Science (HICSS). doi:10.1109/HICSS.2012.421, pp 4168–4177 Gousios G (2013) The GHTorrent dataset and tool suite. In: Proceedings of the 10th Working Conference on Mining Software Repositories, MSR’13, pp 233–236. http://dl.acm.org/citation.cfm?id=2487085.2487132 Greiler M, Herzig K, Czerwonka J (2015) Code ownership and software quality: A replication study. In: Proceedings of the 12th Working Conference on Mining Software Repositories, MSR ’15, pp. 2–12. IEEE Press, Piscataway, NJ, USA. http://dl.acm.org.library.sutd.edu.sg:2048/citation.cfm?id=2820518.2820522 Guzman E, Azócar D, Li Y (2014) Sentiment analysis of commit comments in GitHub: An empirical study. In: MSR, pp 352–355, ACM, New York, NY, USA Guzman E, Bruegge B (2013) Towards emotional awareness in software development teams. In: Joint Meeting on Foundations of Software Engineering, pp 671–674, ACM, New York, NY, USA Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The Weka data mining software: An upyear. SIGKDD Explor Newsl 11(1):10–18 Honkela T, Izzatdust Z, Lagus K (2012) Text mining for wellbeing: Selecting stories using semantic and pragmatic features. In: Artificial Neural Networks and Machine Learning, Part II, LNCS, vol 7553. Springer, pp 467–474 Howard MJ, Gupta S, Pollock LL, Vijay-Shanker K (2013) Automatically mining software-based, semantically-similar words from comment-code mappings. In: Zimmermann T, Penta MD, Kim S (eds) MSR, pp 377–386. IEEE Computer Society Hubert L, Arabie P (1985) Comparing partitions. J Classif 2(1):193–218. doi:10.1007/BF01908075 Islam MR, Zibran MF (2016) Towards understanding and exploiting developers’ emotional variations in software engineering. In: 2016 IEEE 14th International Conference on Software Engineering Research, Management and Applications (SERA), pp 185–192. doi:10.1109/SERA.2016.7516145 Jongeling R, Datta S, Serebrenik A (2015) Choosing your weapons: On sentiment analysis tools for software engineering research. In: ICSME, pp 531–535. IEEE. doi:10.1109/ICSM.2015.7332508 Konietschke F, Hothorn LA, Brunner E (2012) Rank-based multiple test procedures and simultaneous confidence intervals. Electronic Journal of Statistics 6:738–759 Kucuktunc O, Cambazoglu BB, Weber I, Ferhatosmanoglu H (2012) A Large-scale Sentiment Analysis for Yahoo! Answers. In: Proceedings of the Fifth ACM International Conference on Web Search and Data Mining, WSDM ’12, pp 633–642. ACM, New York, NY, USA. doi:10.1145/2124295.2124371 Lanza M, Di Penta M, Xie T (2012) (eds.): 9th IEEE Working Conference of Mining Software Repositories, MSR 2012, June 2-3, 2012, Zurich, Switzerland. IEEE Computer Society. http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=6220358 Leopairote W, Surarerks A, Prompoon N (2013) Evaluating software quality in use using user reviews mining. In: 2013 10th International Joint Conference on Computer Science and Software Engineering (JCSSE), pp 257–262. doi:10.1109/JCSSE.2013.6567355 Lewis DD, Gale WA (1994) A sequential algorithm for training text classifiers. In: Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’94, pp. 3–12. Springer-Verlag New York, Inc., New York, NY, USA. http://dl.acm.org.dianus.libr.tue.nl/citation.cfm?id=188490.188495 Li TH, Liu R, Sukaviriya N, Li Y, Yang J, Sandin M, Lee J (2014) Incident ticket analytics for it application management services. In: 2014 IEEE International Conference on Services Computing (SCC), pp 568–574. doi:10.1109/SCC.2014.80 Lindsey MR (2011) What went wrong?: Negative results from VoIP service providers. In: Proceedings of the 5th International Conference on Principles, Systems and Applications of IP Telecommunications, IPTcomm ’11, pp 13:1–13:3. ACM, New York, NY, USA. doi:10.1145/2124436.2124453 Linstead E, Baldi P (2009) Mining the coherence of GNOME bug reports with statistical topic models. In: Godfrey MW, Whitehead J (eds) Proceedings of the 6th International Working Conference on Mining Software Repositories, MSR 2009 (Co-located with ICSE), Vancouver, BC, Canada, May 16–17, 2009, Proceedings, pp 99–102. IEEE Computer Society. doi:10.1109/MSR.2009.5069486 Martie L, Palepu VK, Sajnani H, Lopes CV Trendy bugs: Topic trends in the android bug reports. In: Lanza et al. [43], pp 120–123. doi:10.1109/MSR.2012.6224268 Mende T (2010) Replication of defect prediction studies: Problems, pitfalls and recommendations. In: Proceedings of the 6th International Conference on Predictive Models in Software Engineering, PROMISE ’10, pp 5:1–5:10. ACM, New York, NY, USA. doi:10.1145/1868328.1868336 Mishne G, Glance NS (2006) Predicting movie sales from blogger sentiment. In: AAAI Spring Symposium: Computational Approaches to Analyzing Weblogs, pp 155–158 Mohammad SM, Kiritchenko S, Zhu X (2013) NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets. arXiv:1308.6242[cs] Murgia A, Tourani P, Adams B, Ortu M (2014) Do developers feel emotions? an exploratory analysis of emotions in software artifacts. In: MSR, pp 262-271, ACM, New York, NY, USA Novielli N, Calefato F, Lanubile F (2015) The challenges of sentiment detection in the social programmer ecosystem. In: Proceedings of the 7th International Workshop on Social Software Engineering, SSE 2015, pp 33–40. ACM, New York, NY, USA. doi:10.1145/2804381.2804387 Ortu M, Adams B, Destefanis G, Tourani P, Marchesi M, Tonelli R (2015) Are bullies more productive? empirical study of affectiveness vs. issue fixing time. In: MSR Ortu M, Destefanis G, Adams B, Murgia A, Marchesi M, Tonelli R (2015) The JIRA repository dataset: Understanding social aspects of software development. In: Proceedings of the 11th International Conference on Predictive Models and Data Analytics in Software Engineering, PROMISE ’15, pp 1:1–1:4. ACM, New York, NY, USA. doi:10.1145/2810146.2810147 Ortu M, Destefanis G, Kassab M, Counsell S, Marchesi M, Tonelli R (2015) Would you mind fixing this issue? - an empirical analysis of politeness and attractiveness in software developed using agile boards. In: Lassenius C, Dingsøyr T, Paasivaara M (eds) Agile Processes, in Software Engineering, and Extreme Programming - 16th International Conference, XP 2015, Helsinki, Finland, May 25–29, 2015, Proceedings, Lecture Notes in Business Information Processing, vol 212. Springer, pp 129–140. doi:10.1007/978-3-319-18612-2_11 Pak A, Paroubek P (2010) Twitter Based System: Using Twitter for Disambiguating Sentiment Ambiguous Adjectives. In: Proceedings of the 5th International Workshop on Semantic Evaluation, SemEval ’10, pp. 436–439. Association for Computational Linguistics, Stroudsburg, PA, USA. http://dl.acm.org/citation.cfm?id=1859664.1859761 Pang B, Lee L (2007) Opinion mining and sentiment analysis. Found Trends Inf Retr 2(1-2):1–135 Pang B, Lee L, Vaithyanathan S (2002) Thumbs Up?: sentiment classification using machine learning techniques. In: Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing - Volume 10, EMNLP ’02, pp 79–86. Association for Computational Linguistics, Stroudsburg, PA, USA. doi:10.3115/1118693.1118704 Panichella S, Sorbo AD, Guzman E, Visaggio CA, Canfora G, Gall HC (2015) How can I improve my app? classifying user reviews for software maintenance and evolution. In: ICSME. IEEE, pp 281–290 Pletea D, Vasilescu B, Serebrenik A (2014) Security and emotion: Sentiment analysis of security discussions on GitHub. In: MSR. ACM, New York, NY, USA, pp 348–351. doi:10.1145/2597073.2597117 Pritchard P (1984) Some negative results concerning prime number generators. Commun ACM 27(1):53–57. doi:10.1145/69605.357970 Rand WM (1971) Objective criteria for the evaluation of clustering methods. J Am Stat Assoc 66(336):846–850 Rousinopoulos AI, Robles G, González-Barahona JM (2014) Sentiment analysis of Free/Open Source developers: preliminary findings from a case study. Revista Eletrônica de Sistemas de Informação 13(2):6:1–6:21 Santos JM, Embrechts M (2009) On the use of the adjusted rand index as a metric for evaluating supervised classification. In: International Conference on Artificial Neural Networks, LNCS, vol 5769. Springer, pp 175–184 Schröter A, Aranda J, Damian D, Kwan I (2012) To talk or not to talk: factors that influence communication around changesets. In: Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work, CSCW ’12, pp 1317–1326. ACM, New York, NY, USA. doi:10.1145/2145204.2145401 Sfetsos P, Adamidis P, Angelis L, Stamelos I, Deligiannis I (2012) Investigating the impact of personality and temperament traits on pair programming: a controlled experiment replication. In: 2012 Eighth International Conference on the Quality of Information and Communications Technology (QUATIC), pp 57–65. doi:10.1109/QUATIC.2012.36 Sheskin DJ (2007) Handbook of parametric and nonparametric statistical procedures, 4 edn. Chapman & Hall Shihab E, Kamei Y, Bhattacharya P (2012) Mining challenge 2012: the Android platform. In: MSR, pp 112–115 Shull FJ, Carver JC, Vegas S, Juristo N (2008) The role of replications in empirical software engineering. Empir Softw Eng 13(2):211–218. doi:10.1007/s10664-008-9060-1 Socher R, Perelygin A, Wu J, Chuang J, Manning CD, Ng A, Potts C (2013) Recursive deep models for semantic compositionality over a sentiment treebank. In: Empirical Methods in Natural Language Processing, pp 1631–1642. Ass. for Comp. Linguistics Sun X, Li B, Leung H, Li B, Li Y (2015) MSR4SM: Using topic models to effectively mining software repositories for software maintenance tasks. Inf Softw Technol 66:1–12. doi:10.1016/j.infsof.2015.05.003. http://www.sciencedirect.com/science/article/pii/S0950584915001007 Täht D (2014) The value of repeatable experiments and negative results: - a journey through the history and future of aqm and fair queuing algorithms. In: Proceedings of the 2014 ACM SIGCOMM Workshop on Capacity Sharing Workshop, CSWS ’14, pp 1–2. ACM, New York, NY, USA. doi:10.1145/2630088.2652480 Thelwall M, Buckley K, Paltoglou G (2012) Sentiment strength detection for the social web. J Am Soc Inf Sci Technol 63(1):163–173 Thelwall M, Buckley K, Paltoglou G, Cai D, Kappas A (2010) Sentiment in short strength detection informal text. J Am Soc Inf Sci Technol 61(12):2544–2558 Tonella P, Torchiano M, Du Bois B, Systä T (2007) Empirical studies in reverse engineering: State of the art and future trends. Empir Softw Eng 12(5):551–571. doi:10.1007/s10664-007-9037-5 Tourani P, Jiang Y, Adams B (2014) Monitoring sentiment in open source mailing lists: exploratory study on the apache ecosystem. In: Proceedings of 24th Annual International Conference on Computer Science and Software Engineering, CASCON ’14, pp 34–44. IBM Corp., Riverton, NJ, USA. http://dl.acm.org/citation.cfm?id=2735522.2735528 Tukey JW (1951) Quick and dirty methods in statistics, part II, Simple analysis for standard designs. In: American Society for Quality Control, pp 189–197 Tumasjan A, Sprenger TO, Sandner PG, Welpe IM (2010) Predicting elections with twitter: What 140 characters reveal about political sentiment. In: International AAAI Conference on Weblogs and Social Media, pp 178–185 van Rijsbergen CJ (1979) Information Retrieval, 2nd edn. Butterworth-Heinemann, Newton, MA, USA Vasilescu B, Filkov V, Serebrenik A (2013) StackOverflow and github: associations between software development and crowdsourced knowledge. In: 2013 International Conference on Social Computing (SocialCom), pp 188–195. doi:10.1109/SocialCom.2013.35 Vasilescu B, Serebrenik A, van den Brand MGJ (2011) By no means: a study on aggregating software metrics. In: Concas G, Tempero ED, Zhang H, Penta MD (eds) Proceedings of the 2nd International Workshop on Emerging Trends in Software Metrics, WETSoM 2011, Waikiki, Honolulu, HI, USA, May 24, 2011. ACM, pp 23–26. doi:10.1145/1985374.1985381 Vasilescu B, Serebrenik A, Goeminne M, Mens T (2013) On the variation and specialisation of workload – a case study of the Gnome ecosystem community. Empir Softw Eng 19(4):955–1008. doi:10.1007/s10664-013-9244-1 Viera AJ, Garrett JM (2005) Understanding interobserver agreement: the kappa statistic. Fam Med 37(5):360–363 Vivian R, Tarmazdi H, Falkner K, Falkner N, Szabo C (2015) The development of a dashboard tool for visualising online teamwork discussions. In: Proceedings of the 37th International Conference on Software Engineering - Volume 2, ICSE ’15, pp 380–388. IEEE Press, Piscataway, NJ, USA. http://dl.acm.org/citation.cfm?id=2819009.2819070 Wang S, Lo D, Vasilescu B, Serebrenik A (2014) EnTagRec: An enhanced tag recommendation system for software information sites. In: ICSME. IEEE, pp 291–300 Wilcoxon F (1945) Individual comparisons by ranking methods. Biom Bull 1 (6):80–83 Wilson T, Wiebe J, Hoffmann P (2005) Recognizing contextual polarity in phrase-level sentiment analysis. In: Human Language Technology and Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Stroudsburg, PA, USA, pp 347–354 Yu HF, Ho CH, Juan YC, Lin CJ (2013) Libshorttext: A library for short-text classification and analysis. Tech. rep., Technical Report. http://www.csie.ntu.edu.tw/~cjlin/papers/libshorttext.pdf Yu Y, Wang H, Yin G, Wang T (2016) Reviewer recommendation for pull-requests in github: What can we learn from code review and bug assignment?. Inf Softw Technol 74:204–218. doi:10.1016/j.infsof.2016.01.004. http://www.sciencedirect.com/science/article/pii/S0950584916000069 Zimmerman DW, Zumbo BD (1992) Parametric alternatives to the Student t test under violation of normality and homogeneity of variance. Percept Mot Skills 74(31):835–844