Mining Supervised Classification Performance Studies: A Meta-Analytic Investigation
Tóm tắt
There have been many comparative studies of classification methods in which real datasets are used as a gauge to assess the relative performance of the methods. Since these comparisons often yield inconclusive or limited results on how methods perform, it is often believed that a broader approach combining these studies would shed some light on this difficult question. This paper describes such an attempt: we have sampled the available literature and created a dataset of 5807 classification results. We show that one of the possible ways to analyze the resulting data is an overall assessment of the classification methods, and we present methods for that particular aim. The merits and demerits of such an approach are discussed, and conclusions are drawn which may assist future research: we argue that the current state of the literature hardly allows large-scale investigations.
Tài liệu tham khảo
BATCHELOR, B. G. and HAND, D. J. (1976), “A Pattern Recognition Competition”, in Proceedings of the Third International Joint Conference on Pattern Recognition, San Diego, 1976.
BERTHOLD, M.R. and DIAMOND, J. (1998), “Constructive Training of Probabilistic Neural Networks”, Neurocomputing, 19:167–183.
BLUE, J.L., CANDELA, G.T., GROTHER, P.J., CHELLAPPA, R., and WILSON, C.L. (1994), “Evaluation of Pattern Classifiers for Fingerprint and OCR Applications”, Pattern Recognition, 4:485–501.
BRADLEY, R.A. and TERRY, M.E. (1952), “Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons”, Biometrika, 39:324–345.
BRAZDIL, P.B., SOARES, C., and PINTO DA COSTA, J. (2003), “Ranking Learning Algorithms: Using IBL and Meta-Learning on Accuracy and Time Results”, Machine Learning, 50:251–277.
COLLETT, D. (2002), Modelling Binary Data (2nd ed.), London: Chapman and Hall.
CURRAM, S.P. and MINGERS, J. (1994), “Neural Networks, Decision Tree Induction and Discriminant Analysis”, Journal of Operational Research Society, 45:440–450.
DIETTERICH, T.G. (2000), “An Experimental Comparison of Three Methods for Constructing Ensembles of Decisions Trees: Bagging, Boosting, and Randomization”, Machine Learning, 40:139–157.
DUIN, R.P.W. (1996), “A Note on Comparing Classifiers”, Pattern Recognition Letters, 17:529–536.
EKLUND, P.W. and HOANG, A. (2002), “A Performance Survey of Public Domain Supervised Machine Learning Algorithms”, http://citeseer.nj.nec.com/142129.html.
FISHER, R.A. (1936), “The Use of Multiple Measurements in Taxonomic Problems”, Annals of Eugenics, 7:179–188.
FUKUNAGA, K. (1990), Introduction to Statistical Pattern Recognition, San Diego: Academic Press.
HAND, D.J. (2004), “Academic Obsessions and Classification Realities: Ignoring Practicalities in Supervised Classification”, in Classification, Clustering and Data Mining Applications, eds. B. Banks, L. House, F. R. McMorris, P. Arabie, and W. Gaul, Berlin: Springer, pp. 209–232.
HAND, D.J. (1981), Discrimination and Classification, Chichester: Wiley.
HAND, D.J. (1997), Construction and Assessment of Classification Rules, Chichester: Wiley.
HAND, D.J., MANNILA, H., and SMYTH, P. (2001), Principles of Data Mining, Cambridge MA: MIT Press.
HASTIE, T., TIBSHIRANI, R., and FRIEDMAN, J. (2001), The Elements of Statistical Learning Theory, New York: Springer.
HOOPER, P.M. (1999), “Reference Point Logistic Classification”, Journal of Classification, 16(1):91–116.
JAMAIN, A. (2004), “Meta-analysis of Classification Methods”, PhD thesis, Department of Mathematics, Imperial College, London.
JAMAIN, A. and HAND, D.J. (2005), “The Naive Bayes Mystery: a Classification Detective Story”, Pattern Recognition Letters, 26:1752–1760.
KLEINBERG, E.M. (2000), “On the Algorithmic Implementation of Stochastic Discrimination”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(5): 473–490.
LIM, T., LOH, W., and SHIH, Y. (2000), “A Comparison of Prediction Accuracy, Complexity, and Training Time of Thirty-Three Old and New Classification Algorithms”, Machine Learning, 40:203–228.
MCLACHLAN, G.J. (1992), Discriminant Analysis and Statistical Pattern Recognition, New York: Wiley.
METAL CONSORTIUM (2002), “Esprit Project METAL (#26.357)”, http://www.metalkdd.org.
MICHIE, D., SPIEGELHALTER, D.J., and TAYLOR, C.C. (1994), Machine Learning, Neural and Statistical Classification, New York: Ellis Horwood.
MITCHELL, T.M. (1997), Machine Learning, New York: McGraw-Hill.
RASMUSSEN, C.E., NEAL, R.M., HINTON, G.E., VAN CAMP, D., REVOW, M., GHAHRAMANI, Z., KUSTRA, R., and TIBSHIRANI, R. (1996), “DELVE, Data for Evaluating Learning in Valid Experiments”, http://www.cs.toronto.edu/~delve/.
RENDELL, L. and SESHU, R. (1990), “Learning Hard Concepts Through Constructive Induction”, Computational Intelligence, 6:247–270.
RIPLEY, B.D. (1996), Pattern Recognition and Neural Networks, Cambridge: Cambridge University Press.
SARGENT, D.J. (2001), “Comparison of Artificial Neural Networks with Other Statistical Approaches”, Cancer, 91:1636–42.
SCHIAVO, R.A. and HAND, D.J. (2000), “Ten More Years of Error Rate Research”, International Statistical Review, 68(3):295–310.
SOHN, S.Y. (1999), “Meta-analysis of Classification Algorithms for Pattern Recognition”, IEEE Transactions on Pattern Recognition and Machine Intelligence, 21(11):1137–1144.
VAN DER LINDEN, W.J. and HAMBLETON, R.K. (1997), Handbook of Modern Item Response Theory, New York: Springer-Verlag.
WEBB, A. (2002), Statistical Pattern Recognition (2nd ed.), London: Arnold.
ZARNDT, F. (1995), “A Comprehensive Case Study: an Examination of Machine Learning and Connectionnist Algorithms”, http://citeseer.nj.nec.com/481595.html.