Evaluating the performance of open source software projects using data envelopment analysis

Emerald - Tập 16 Số 5 - Trang 449-462 - 2008
Barry Wray1, Richard Mathieu2
1Information Systems and Operations Management, UNC Wilmington, Wilmington, North Carolina, USA
2Department of CIS and Management Science, James Madison University, Harrisonburg, Virginia, USA

Tóm tắt

PurposeThe purpose of this paper is to develop and test a model of the relative performance of open source software (OSS) projects.Design/methodology/approachThis paper evaluates the relative performance of OSS projects by evaluating multiple project inputs and multiple project outputs by using a data envelopment analysis (DEA) model. The DEA model produces an efficiency score for each project based on project inputs and outputs. The method of producing an efficiency score is based on the convex envelopment technology structure. The efficiency measure quantifies a “distance” to an efficient frontier.FindingsThe DEA model produced an index of corresponding intensities linking an inefficient project to its benchmark efficient project(s). The inefficiency measures produced an ordering of inefficient projects. Eight projects were found to be “efficient” and used as benchmarking projects.Research limitations/implicationsThis research is limited to only security‐based OSS projects. Future research on other areas of OSS projects is warranted.Practical implicationsThe result of this research is a practical model that can be used by OSS project developers to evaluate the relative performance of their projects and make resource decisions.Originality/valueThis research extends the work of previous studies that have examined the relative performance of software development projects in a traditional development environment. As a result of this research, OSS projects can now be adequately benchmarked and evaluated according to project performance. An OSS project manger can effectively use these results to critically evaluate resources for their project and judge the relative efficiency of the resources.

Tài liệu tham khảo

Banker, R.D. and Kemerer, C.F. (1989), “Scale economies in new software development”, IEEE Transactions on Software Development, Vol. 15 No. 10, pp. 1199‐205. Banker, R.D., Datar, S.M. and Kemerer, C.F. (1991), “A model to evaluate variables impacting the production of software maintenance projects”, Management Science, Vol. 37 No. 1, pp. 1‐8. Charnes, A., Rousseau, J. and Semple, J. (1996), “Sensitivity and stability of efficiency classifications in data envelopment analysis”, Journal of Productivity Analysis, Vol. 7 No. 1, pp. 5‐18. Charnes, A., Cooper, W.W., Golany, B., Seiford, L. and Stutz, J. (1985), “Foundations of data envelopment analysis for Pareto‐Koopmans efficient empirical production functions”, Journal of Econometrics, Vol. 30, pp. 91‐107. Colbert, A., Levary, R. and Shaner, M. (2000), “Determining the relative efficiency of MBA programs using DEA”, European Journal of Operational Research, Vol. 125, pp. 656‐69. Crowston, K., Annabi, H. and Howison, J. (2003), “Defining open source software project success”, paper presented at 24th International Conference on Information Systems (ICIS), pp. 327‐40. Elam, J. (1991), “Evaluating the efficiency of IS organizations using data envelope analysis”, Proceedings of the International Function Point Users Group 1991 Fall Conference, October. Farrell, M. (1957), “The measurement of productive efficiency”, Journal of the Royal Statistical Society, Series A, Vol. 120 No. 3, pp. 253‐90. Feller, J. and Fitzgerald, B. (2002), Understanding Open Source Software Development, Addison Wesley, London. Flitman, A. (2003), “Towards meaningful benchmarking of software development team productivity”, Benchmarking: An International Journal, Vol. 10 No. 4, pp. 382‐99. Greenemeier, L. (2005), “OPEN‐SOURCE EXUBERANCE – hot‐shot entrepreneurs, VC money, even paradigm shifts. Dot‐coms revisited? No, it's today's flood of open‐source startups”, Information Week, July 11. Hahn, J. and Zhang, C. (2005), “An exploratory study of open source projects from a project management perspective”, paper presented at MIS Research Workshop, Purdue University, West Lafayette, IN. Jorgensen, N. (2001), “Putting it all in the trunk: incremental software development in the FreeBSD open source project”, Information Systems Journal, Vol. 11 No. 4, pp. 321‐36. Kalina, I. and Czyzycki, A. (2005), “The ins and outs of open source”, Consulting to Management, Vol. 16 No. 3, pp. 41‐7. Kogut, B. and Metiu, A. (2001), “Open source software development and distributed innovation”, Oxford Review of Economic Policy, Vol. 17 No. 2, pp. 248‐64. Lacy, S. (2005), “Open source: now it's an ecosystem”, BusinessWeek Online, October, p. 3. Mogull, R. and Girard, J. (2006), “Assessment methodology for open‐source security testing tools”, Gartner Research, (ID G00144478), December. Mookhey, K.K. (2004), “Open source tools for security and control assessment”, Information Systems Control Journal, Vol. 1, pp. 39‐44. Paradi, J.C., Reese, D.N. and Rosen, D. (1997), “Applications of DEA to measure the efficiency of software production at two large Canadian banks”, Annals of Operations Research, Vol. 73, pp. 91‐115. Payne, C. (2002), “On the security of open source software”, Information Systems Journal, Vol. 12 No. 1, pp. 61‐78. Salkever, A. (2001), “Is open‐source security software safe?”, BusinessWeek Online, December, p. 12. Scacchi, W. (2002), “Is open source software development faster, better and cheaper than software engineering”, paper presented at 2nd ICSE Workshop on Open Source Software Engineering, Orlando, FL, May. Scacchi, W. (2004), “Free and open source development practices in the game community”, IEEE Software, Vol. 21 No. 1, pp. 59‐66. Scacchi, W., Feller, J., Fitzgerald, B., Hissam, S. and Lakhani, K. (2006), “Understanding free/open source software development processes”, Software Process Improvement and Practice, Vol. 11, pp. 95‐105. Scheel, H. (2000), EMS: Efficiency Measurement System User's Manual, available at: www.wiso.uni‐dortmund.de/lsfg/or/scheel/ems/. Stamelos, I., Angelis, L., Oikonomou, A. and Bleris, G. (2002), “Code quality analysis in open source software development”, Information Systems Journal, Vol. 12, pp. 43‐60. Stensrud, E. and Myrtveit, I. (2003), “Identifying high performance ERP projects”, IEEE Transactions on Software Engineering, Vol. 29 No. 5, pp. 398‐415. Yang, Z. and Paradi, J.C. (2004), “DEA evaluation of Y2K software retrofit program”, IEEE Transactions on Engineering Management, Vol. 51 No. 3, pp. 279‐87.
Thông tin
Thông tin xuất bản

Emerald

Tập 16 Số 5

449-462

Thông tin tác giả