Fast online algorithm for nonlinear support vector machines and other alike models
Tóm tắt
Paper presents a unique novel online learning algorithm for eight popular nonlinear (i.e., kernel), classifiers based on a classic stochastic gradient descent in primal domain. In particular, the online learning algorithm is derived for following classifiers: L1 and L2 support vector machines with both a quadratic regularizer w
t
w and the l
1 regularizer |w|1; regularized huberized hinge loss; regularized kernel logistic regression; regularized exponential loss with l
1 regularizer |w|1 and Least squares support vector machines. The online learning algorithm is aimed primarily for designing classifiers for large datasets. The novel learning model is accurate, fast and extremely simple (i.e., comprised of few coding lines only). Comparisons of performances of the proposed algorithm with the state of the art support vector machine algorithm on few real datasets are shown.
Tài liệu tham khảo
Vapnik, V.N., Chervonenkis, A.Y., On the uniform convergence of relative frequencies of events to their probabilities. (In Russian), Doklady Akademii Nauk USSR, 1968, (4), pp. 181–186.
Cortes, C., Vapnik, V.N., Support Vector Networks. Machine Learning 1995, (20), pp. 273–297.
Vapnik, V.N., The Nature of Statistical Learning Theory, Springer, 1995.
Kecman, V., Learning and Soft Computing, Support Vector Machines, Neural Networks, and Fuzzy Logic Models, The MIT Press, 2001.
Schölkopf, B., Smola, A. J., Learning with Kernels, The MIT Press, 2002.
Kivinen J., Smola A.J., Williamson R. C., Large Margin Classification for Drifting Targets, Neural Information Processing Systems, 2002.
Shalev-Shwartz, S., Singer, Y., Srebro, N., Pegasos: Primal estimated subgradient solver for SVM, Proc. 24th Intl. Conf. on ML (ICML’07), ACM, 2007, pp. 807–814.
Shalev-Shwartz, S., Ben-David, S., Understanding Machine Learning—From Theory to Algorithms, New York: Cambridge University Press, 2014.
Bottou, L., Large-Scale Machine Learning with Stochastic Gradient Descent, Proc. 19th Int. Conf. on Comp. Stats., COMPSTAT’2010, Springer, 2010, pp. 177–187.
Herbrich, R., Learning Kernel Classifiers, Theory and Algorithms, The MIT Press, 2002.
Collobert, R., Bengio, S., Links between Perceptrons, MLPs and SVMs, Proc. of the 21st Intl. Conf. on Machine Learning, Banff, Canada, 2004.
Steinwart, I., Christmann, A., Support Vector Machines, Springer, 2008.
Zhu, J., Hastie, T., Kernel Logistic Regression and the Import Vector Machine, Journal of Computational and Graphical Statistics, 2005, V. 14, No: 1, pp. 185–20.
Melki, G., Kecman, V., Speeding-Up On-Line Training of L1 Support Vector Machines, Proc. of IEEE SoutheastConf, Norfolk, VA, 2016.
Kecman, V., Fast online algorithm for nonlinear support vector machines and other alike models, Seminar at VCU, 2016.
Platt, J.C., Sequential Minimal Optimization, A Fast Algorithm for Training Support Vector Machines, Microsoft Research, Technical Report MSR-TR-98-14, 1998.
Chang, C-C. and Lin, C-J., “LIBSVM: A library for support vector machines”. In: ACM Transactions on Intelligent Systems and Technology 2, Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm, 2011, 27:1–27:27.
Kecman, V., Huang, T.-M., Vogt, M., Iterative Single Data Algorithm for Training Kernel Machines from Huge Data Sets: Theory and Performance, In Support Vector Machines: Theory and Applications. Ed. Lipo Wang, Berlin: Springer-Verlag, 2005, pp. 255–274.
Huang, T.-M., Kecman, V., Kopriva, I., Kernel Based Algorithms for Mining Huge Data Sets, Supervised, Semisupervised, and Unsupervised Learning, Springer-Verlag, Berlin, Heidelberg, see http://www.learning-fromdata. com 2005.
http://www.mathworks.com/help/stats/fitcsvm.html.