Multiclass classification by Min–Max ECOC with Hamming distance optimization

The Visual Computer - Tập 39 - Trang 3949-3961 - 2022
Gábor Szűcs1
1Department of Telecommunications and Media Informatics, Budapest University of Technology and Economics, Budapest, Hungary

Tóm tắt

Two questions often arise in the field of the ensemble in multiclass classification problems, (i) how to combine base classifiers and (ii) how to design possible binary classifiers. Error-correcting output codes (ECOC) methods answer these questions, but they focused on only the general goodness of the classifier. The main purpose of our research was to strengthen the bottleneck of the ensemble method, i.e., to minimize the largest values of two types of error ratios in the deep neural network-based classifier. The research was theoretical and experimental, the proposed Min–Max ECOC method suggests a theoretically proven optimal solution, which was verified by experiments on image datasets. The optimal solution was based on the maximization of the lowest value in the Hamming matrix coming from the ECOC matrix. The largest ECOC matrix, the so-called full matrix is always a Min–Max ECOC matrix, but smaller matrices generally do not reach the optimal Hamming distance value, and a recursive construction algorithm was proposed to get closer to it. It is not easy to calculate optimal values for large ECOC matrices, but an interval with upper and lower limits was constructed by two theorems, and they were proved. Convolutional Neural Networks with Min–Max ECOC matrix were tested on four real datasets and compared with OVA (one versus all) and variants of ECOC methods in terms of known and two new indicators. The experimental results show that the suggested method surpasses the others, thus our method is promising in the ensemble learning literature.

Tài liệu tham khảo

Ahmed, S.A.A., Zor, C., Awais, M., Yanikoglu, B., Kittler, J.: Deep convolutional neural network ensembles using ECOC. IEEE Access 9, 86083–86095 (2021)

Allwein, E.L., Schapire, R.E., Singer, Y.: Reducing multiclass to binary: a unifying approach for margin classifiers. J. Mach. Learn. Res. 1, 113–141 (2001)

Alshdaifat, E.A., Coenen, F., Dures, K.: A directed acyclic graph based approach to multi-class ensemble classification. In: International Conference on Innovative Techniques and Applications of Artificial Intelligence, pp. 43–57. Springer, Cham (2015)

Alvear-Sandoval, R.F., Sancho-Gómez, J.L., Figueiras-Vidal, A.R.: On improving CNNs performance: the case of MNIST. Inf. Fusion 52, 106–109 (2019). https://doi.org/10.1016/j.inffus.2018.12.005

Bagheri, M.A., Gao, Q., Escalera, S. Generic subclass ensemble: a novel approach to ensemble classification. In: 2014 22nd International Conference on Pattern Recognition, pp. 1254–1259. IEEE (2014)

Bora, M.B., Daimary, D., Amitab, K., Kandar, D.: Handwritten character recognition from images using CNN-ECOC. Procedia Comput. Sci. 167, 2403–2409 (2020)

Chaladze, G., Kalatozishvili, L.: Linnaeus 5 dataset for machine learning (2017)

Chen, S.G., Wu, X.J.: Multiple birth least squares support vector machine for multi-class classification. Int. J. Mach. Learn. Cybern. 8(6), 1731–1742 (2017)

Cheng, Y., Liu, Y., Zhu, X., Li, S.: A multiclassification method for iris data based on the Hadamard error correction output code and a convolutional network. IEEE Access 7, 145235–145245 (2019)

D’Ambrosio, R., Iannello, G., Soda, P.: Softmax regression for ecoc reconstruction. In: International Conference on Image Analysis and Processing, pp. 682–691. Springer, Berlin, Heidelberg (2013)

Dietterich, T.G., Bakiri, G.: Solving multiclass learning problems via error-correcting output codes. J. Artif. Intell. Res. 2, 263–286 (1994)

Dong, X., Yu, Z., Cao, W., Shi, Y., Ma, Q.: A survey on ensemble learning. Front. Comp. Sci. 14(2), 241–258 (2020)

Gao, X., He, Y., Zhang, M., Diao, X., Jing, X., Ren, B., Ji, W.: A multiclass classification using one-versus-all approach with the differential partition sampling ensemble. Eng. Appl. Artif. Intell. 97, 104034 (2021)

García-Pedrajas, N., Ortiz-Boyer, D.: An empirical study of binary classifier fusion methods for multiclass classification. Inf. Fusion 12(2), 111–130 (2011)

Gogić, I., Manhart, M., Pandžić, I.S., Ahlberg, J.: Fast facial expression recognition using local binary features and shallow neural networks. Vis. Comput. 36(1), 97–112 (2020). https://doi.org/10.1007/s00371-018-1585-8

Guo, H., Liu, Y., Yang, D., Zhao, J.: Offline handwritten Tai Le character recognition using ensemble deep learning. Vis. Comput. (2021). https://doi.org/10.1007/s00371-021-02230-2

Hastie, T., Tibshirani, R.: Classification by pairwise coupling. Ann. Stat. 26(2), 451–471 (1998)

He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90

He, Q., Li, X., Kim, D.N., Jia, X., Gu, X., Zhen, X., Zhou, L.: Feasibility study of a multi-criteria decision-making based hierarchical model for multi-modality feature and multi-classifier fusion: applications in medical prognosis prediction. Inf. Fusion 55, 207–219 (2020)

Klimo, M., Lukáč, P., Tarábek, P.: Deep neural networks classification via binary error-detecting output codes. Appl. Sci. 11(8), 3563 (2021)

Krawczyk, B., Woźniak, M., Herrera, F.: On the usefulness of one-class classifier ensembles for decomposition of multi-class problems. Pattern Recognit. 48(12), 3969–3982 (2015)

Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical report, University of Toronto (2009)

Lachaize, M., Le Hégarat-Mascle, S., Aldea, E., Maitrot, A., Reynaud, R.: Evidential split-and-merge: application to object-based image analysis. Int. J. Approx. Reason. 103, 303–319 (2018)

Lei, L., Song, Y.: Weighted decoding for the competence reliability problem of ECOC multiclass classification. Comput. Intell. Neurosci. 2021, Article ID 5583031, 11 pp (2021). https://doi.org/10.1155/2021/5583031

Liu, K.H., Zeng, Z.H., Ng, V.T.Y.: A hierarchical ensemble of ECOC for cancer classification based on multi-class microarray data. Inf. Sci. 349, 102–118 (2016)

Manikandan, J., Venkataramani, B.: System-on-programmable-chip implementation of diminishing learning based pattern recognition system. Int. J. Mach. Learn. Cybern. 4(4), 347–363 (2013)

Mehra, N., Gupta, S.: Survey on multiclass classification methods. Int. J. Comput. Sci. Inf. Technol. (IJCSIT) 4(4), 572–576 (2013)

Nazari, S., Moin, M.S., Kanan, H.R.: A discriminant binarization transform using genetic algorithm and error-correcting output code for face template protection. Int. J. Mach. Learn. Cybern. 10(3), 433–449 (2019)

Neill, J.O., Bollegala, D.: Error-correcting neural sequence prediction (2019). arXiv preprint arXiv:1901.07002.

Rocha, A., Goldenstein, S.K.: Multiclass from binary: expanding one-versus-all, one-versus-one and ecoc-based approaches. IEEE Trans. Neural Netw. Learn. Syst. 25(2), 289–302 (2013)

Scheidegger, F., Istrate, R., Mariani, G., Benini, L., Bekas, C., Malossi, C.: Efficient image dataset classification difficulty estimation for predicting deep-learning accuracy. Vis. Comput. 37(6), 1593–1610 (2021). https://doi.org/10.1007/s00371-020-01922-5

Shiraishi, Y., Fukumizu, K.: Statistical approaches to combining binary classifiers for multi-class classification. Neurocomputing 74(5), 680–688 (2011)

Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2014). arXiv preprint arXiv:1409.1556

Sobczak, S., Kapela, R., McGuinness, K., Swietlicka, A., Pazderski, D., O’Connor, N.E.: Restricted Boltzmann machine as an aggregation technique for binary descriptors. Vis. Comput. 37, 423–432 (2021)

Stallkamp, J., Schlipsing, M., Salmen, J., Igel. C.: The German Traffic Sign Recognition Benchmark: a multi-class classification competition. In: Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN 2011), pp. 1453–1460 (2011)

Sun, J., Fujita, H., Zheng, Y., Ai, W.: Multi-class financial distress prediction based on support vector machines integrated with the decomposition and fusion methods. Inf. Sci. 559, 153–170 (2021)

Thakkar, A., Chaudhari, K.: Fusion in stock market prediction: a decade survey on the necessity, recent developments, and potential future directions. Inf. Fusion 65, 95–107 (2021)

Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms (2017). arXiv preprint arXiv:1708.07747

Yan, J., Zhang, Z., Lin, K., Yang, F., Luo, X.: A hybrid scheme-based one-vs-all decision trees for multi-class classification tasks. Knowl. Based Syst. 198, 105922 (2020)

Ye, X.N., Liu, K.H., Liong, S.T.: A ternary bitwise calculator based genetic algorithm for improving error correcting output codes. Inf. Sci. 537, 485–510 (2020)

Zadrozny, B., Elkan, C.: Transforming classifier scores into accurate multiclass probability estimates. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 694–699 (2002).

Zhang, H., Hu, Z., Hao, R.: Joint information fusion and multi-scale network model for pedestrian detection. Vis. Comput. 37, 2433–2442 (2021)

Zhou, J.T., Tsang, I.W., Ho, S.S., Müller, K.R.: N-ary decomposition for multi-class classification. Mach. Learn. 108(5), 809–830 (2019)

Zhou, L., Wang, Q., Fujita, H.: One versus one multi-class classification fusion using optimizing decision directed acyclic graph for predicting listing status of companies. Inf. Fusion 36, 80–89 (2017)

Zou, J.Y., Sun, M.X., Liu, K.H., Wu, Q.Q.: The design of dynamic ensemble selection strategy for the error-correcting output codes family. Inf. Sci. (2021). https://doi.org/10.1016/j.ins.2021.04.038

Thông tin
Thông tin xuất bản

The Visual Computer

Tập 39

3949-3961

Thông tin tác giả