Active Learning Query Strategies for Classification, Regression, and Clustering: A Survey

Mitchell T. Machine Learning (1st edition). MacGraw-Hill Education, 1997.

Hu R. Active learning for text classification [Ph.D. Thesis]. Dublin Institute of Technology, 2011.

Tuia D, Ratle F, Pacifici F, Kanevski M F, Emery W J. Active learning methods for remote sensing image classification. IEEE Trans. Geoscience and Remote Sensing, 2009, 47(7-2): 2218-2232.

Guo J, Chen H, Sun Z, Lin Y. A novel method for protein secondary structure prediction using dual-layer SVM and profiles. PROTEINS: Structure, Function, and Bioinformatics, 2004, 54(4): 738-743.

Zhu X. Semi-supervised learning literature survey. Technical Report, University of Wisconsin-Madison, 2008. http://pages.cs.wisc.edu/∼jerryzhu/pub/ssl survey.pdf, Nov. 2019.

Settles B. Active learning literature survey. Technical Report, University of Wisconsin-Madison, 2009. http://apophenia.wdfiles.com/local–files/start/settles active.learning.pdf, Nov. 2019.

Cohn D, Atlas L, Ladner R. Improving generalization with active learning. Machine Learning, 1994, 15(2): 201-221.

Wang M, Hua X S. Active learning in multimedia annotation and retrieval: A survey. ACM Trans. Intelligent Systems and Technology, 2011, 2(2): Article No. 10.

Lewis D D, Catlett J. Heterogeneous uncertainty sampling for supervised learning. In Proc. the 11th Int. Conference on Machine Learning, July 1994, pp.148-156.

Zhu X, Zhang P, Lin X, Shi Y. Active learning from data streams. In Proc. the 7th IEEE Int. Conference on Data Mining, October 2007, pp.757-762.

Zhu X, Zhang P, Lin X, Shi Y. Active learning from stream data using optimal weight classifier ensemble. IEEE Trans. Systems, Man, and Cybernetics, Part B, 2010, 40(6): 1607-1621.

Zliobaite I, Bifet A, Pfahringer, Holmes G. Active learning with drifting streaming data. IEEE Trans. Neural Networks and Learning Systems, 2014, 25(1): 27-39.

Wang P, Zhang P, Guo L. Mining multi-label data streams using ensemble-based active learning. In Proc. the 12th SIAM International Conference on Data Mining, April 2012, pp.1131-1140.

Angluin D. Queries and concept learning. Machine Learning, 1988, 2(4): 319-342.

Wang L, Hu X, Yuan B, Lu J. Active learning via query synthesis and nearest neighbour search. Neurocomputing, 2015, 147: 426-434.

Sun L L, Wang X Z. A survey on active learning strategy. In Proc. the Int. Conference on Machine Learning and Cybernetics, July 2010, pp.161-166.

Fu Y, Zhu X, Li B. A survey on instance selection for active learning. Knowledge and Information Systems, 2012, 35(2): 249-283.

Aggarwal C, Kong X, Gu Q, Han J, Yu P. Active learning: A survey. In Data Classification: Algorithms and Applications, Aggarwal C C (ed.), CRC Press, 2014, pp.571-605.

Lewis D D, Gale W A. A sequential algorithm for training text classifiers. In Proc. the 17th Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, July 1994, pp.3-12.

Atlas L, Cohn D A, Ladner R E. Training connectionist networks with queries and selective sampling. In Proc. the 3rd Annual Conference on Neural Information Processing Systems, November 1989, pp.566-573.

Culotta A, Mccallum A. Reducing labeling effort for structured prediction tasks. In Proc. the 20th National Conference on Artificial Intelligence, July 2005, pp.746-751.

Shannon C E. A mathematical theory of communication. Bell System Technical Journal, 1948, 27(3): 379-423.

Scheffer T, Decomain C, Wrobel S. Active hidden Markov models for information extraction. In Proc. the 4th International Conference on Advances in Intelligent Data Analysis, September 2001, pp.309-318.

Seung H, Opper M, Sompolinsky H. Query by committee. In Proc. the 5th Annual Conference on Computational Learning Theory, July 1992, pp.287-294.

Abe N, Mamitsuka H. Query learning strategies using boosting and bagging. In Proc. the 15th International Conference on Machine Learning, July 1998, pp.1-9.

Melville P, Mooney R J. Diverse ensembles for active learning. In Proc. the 21st Int. Conference on Machine learning, July 2004, Article No. 56.

Muslea I, Minton S, Knoblock C A. Selective sampling with redundant views. In Proc. the 17th National Conference on Artificial Intelligence, July 2000, pp.621-626.

Cortes C, Vapnik V. Support-vector networks. Machine Learning, 1995, 20(3): 273-297.

Kremer J, Pedersen K S, Igel C. Active learning with support vector machines. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 2014, 4(4): 313-326.

Tong S, Koller D. Support vector machine active learning with applications to text classification. Journal of Machine Learning Research, 2002, 2: 45-66.

Vapnik V. An overview of statistical learning theory. IEEE Trans. Neural Networks, 1999, 10(5): 988-999.

Schohn G, Cohn D. Less is more: Active learning with support vector machines. In Proc. the 17th Int. Conference on Machine Learning, June 2000, pp.839-846.

Campbell C, Cristianini N, Smola A. Query learning with large margin classifiers. In Proc. the 17th Int. Conference on Machine Learning, June 2000, pp.111-118.

Indyk P, Motwani R. Approximate nearest neighbors: Towards removing the curse of dimensionality. In Proc. the 30th Annual ACM Symposium on Theory of Computing, May 1998, pp.604-613.

Gionis A, Indyk P, Motwani R. Similarity search in high dimension via hashing. In Proc. the 25th Int. Conference on Very Large Data Bases, September 1999, pp.518-529.

Jain P, Vijayanarasimhan S, Grauman K. Hashing hyperplane queries to near points with applications to large-scale active learning. In Proc. the 24th Annual Conference on Neural Information Processing Systems, December 2010, pp.928-936.

Vijayanarasimhan S, Jain P, Grauman K. Hashing hyperplane queries to near points with applications to large-scale active learning. IEEE Trans. Pattern Analysis and Machine Intelligence, 2014, 36(2): 276-288.

Basri R, Hassner T, Zelnik-Manor L. Approximate nearest subspace search. IEEE Trans. Pattern Analysis and Machine Intelligence, 2011, 33(2): 266-278.

Basri R, Hassner T, Zelnik-Manor L. Approximate nearest subspace search with applications to pattern recognition. In Proc. the 2017 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June 2007.

Wang J, Shen H, Song J, Ji J. Hashing for similarity search: A survey. arXiv:1408.2927, 2014. http://arxiv.org/abs/1408.2927, Nov. 2019.

Settles B, Craven M. An analysis of active learning strategies for sequence labeling tasks. In Proc. the 2008 Conference on Empirical Methods in Natural Language Processing, October 2008, pp.1070-1079.

Wu Y, Kozintsev I, Bouguet J Y, Dulong C. Sampling strategies for active learning in personal photo retrieval. In Proc. the IEEE International Conference on Multimedia and Expo, July 2006, pp.529-532.

Ienco D, Bifet A, Zliobaite I et al. Clustering based active learning for evolving data streams. In Proc. the 16th Int. Conference on Discovery Science, October 2013, pp.79-93.

Brinker K. Incorporating diversity in active learning with support vector machines. In Proc. the 20th Int. Conference on Machine Learning, August 2003, pp.59-66.

Hoi S C H, Jin R, Lyu M R. Large-scale text categorization by batch mode active learning. In Proc. the 15th Int. Conference on World Wide Web, May 2006, pp.633-642.

Hoi S C H, Jin R, Zhu J, Lyu M R. Batch mode active learning and its application to medical image classification. In Proc. the 23rd Int. Conference on Machine Learning, June 2006, pp.417-424.

Xu Z, Akella R, Zhang Y. Incorporating diversity and density in active learning for relevance feedback. In Proc. the 29th Eur. Conf. Inf. Retrieval Research, April 2007, pp.246-257.

Sen P, Namata G, Bilgic M, Getoor L, Galligher B, Eliassi- Rad T. Collective classification in network data. AI Magazine, 2008, 29(3): 93-106.

Neville J, Jensen D. Iterative classification in relational data. In Proc. the AAAI 2000 Workshop on Learning Statistical Models from Relational Data, July 2000, pp.42-49.

Richardson M, Domingos P. Markov logic networks. Machine Learning, 2006, 62(1/2): 107-136.

Bilgic M, Mihalkova L, Getoor L. Active learning for networked data. In Proc. the 27th Int. Conference on Machine Learning, June 2010, pp.79-86.

Wang Z, Ye J. Querying discriminative and representative samples for batch mode active learning. In Proc. the 19th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, August 2013, pp.158-166.

Nguyen H T, Smeulders A. Active learning using preclustering. In Proc. the 21st Int. Conference on Machine Learning, July 2004, Article No. 19.

Huang S J, Jin R, Zhou Z H. Active learning by querying informative and representative examples. IEEE Trans. Pattern Analysis and Machine Intelligence, 2014, 36(10): 1936-1949.

Hoi S C, Jin R, Zhu J, Lyu M R. Semi-supervised SVM batch mode active learning for image retrieval. In Proc. the 2008 IEEE Conference on Computer Vision and Pattern Recognition, June 2008, Article No. 10.

Belkin M, Niyogi P, Sindhwani V. Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. Journal of Machine Learning Research, 2006, 7: 2399-2434.

Du B, Wang Z, Zhang L, Zhang L, Liu W, Shen J, Tao D. Exploring representativeness and informativeness for active learning. IEEE Trans. Cybernetics, 2017, 47(1): 14-26.

Gretton A, Borgwardt K M, RaschM J, Schölkopf B, Smola A. A kernel two-sample test. Journal of Machine Learning Research, 2012, 13: 723-773.

Luo W, Schwing A, Urtasun R. Latent structured active learning. In Proc. the 27th Annual Conference on Neural Information Processing Systems, December 2013, pp.728-736.

Anderson N, Hall P, Titterington D. Two-sample test statistics for measuring discrepancies between two multivariate probability density functions using kernel-based density estimates. Journal of Multivariate Analysis, 1994, 50(1): 41-54.

Wang Z, Fang X, Tao X et al. Multi-class active learning by integrating uncertainty and diversity. IEEE Access, 2018, 6: 22794-22803.

Krempl G, Kottke D, Spiliopoulou M. Probabilistic active learning: Towards combining versatility, optimality and efficiency. In Proc. the 17th Int. Conference on Discovery Science, October 2014, pp.168-179.

Chapelle O, Sch¨olkopf B, Zien A. Semi-Supervised Learning. The MIT Press, 2010.

Krempl G, Kottke D, Lemaire V. Optimised probabilistic active learning (OPAL) — For fast, non-myopic, costsensitive active classification. Machine Learning, 2015, 100(2-3): 449-476.

Settles B, Craven M, Ray S. Multiple-instance active learning. In Proc. the 21st Annual Conference on Neural Information Processing Systems, December 2007, pp.1289-1296.

Roy N, McCallum A. Toward optimal active learning through sampling estimation of error reduction. In Proc. the 18th Int. Conference on Machine Learning, June 2001, pp441-448.

Moskovitch R, Nissim N, Stopel D et al. Improving the detection of unknown computer worms activity using active learning. In Proc. the 30th German Conference on AI, September 2007, pp.489-493.

Fang M, Li Y, Cohn T. Learning how to active learn: A deep reinforcement learning approach. In Proc. the Conference on Empirical Methods in Natural Language Processing, September 2017, pp.595-605.

Liu M, Buntine W, Haffari G. Learning how to actively learn: A deep imitation learning approach. In Proc. the 56th Annual Meeting of the Association for Computational Linguistics, July 2018, pp.1874-1883.

Pang K, Dong M, Wu Y et al. Meta-learning transferable active learning policies by deep reinforcement learning. arXiv:1806.04798, 2008. https://arxiv.org/abs/1806.04798, Nov. 2019.

Bachman P, Sordoni A, Trischler A. Learning algorithms for active learning. In Proc. the 34th Int. Conference on Machine Learning, August 2017, pp.301-310.

Cohn D, Ghahramani Z, Jordan M. Active learning with statistical models. In Proc. the 1994 Annual Conference on Neural Information Processing Systems, December 1994, pp.705-712.

Geman S, Bienenstock E, Doursat R. Neural networks and the bias/variance dilemma. Neural Computation, 1992, 4(1): 1-58.

Schervish M. Theory of Statistics (1st edition). Springer, 1995.

Long B, Chapelle O, Zhang Y, Chang Y, Zheng Y, Tseng B. Active learning for ranking through expected loss optimization. IEEE Trans. Knowledge and Data Engineering, 2015, 27(5): 1180-1191.

Freund Y, Seung H, Shamir E, Tishby N. Selective sampling using the query by committee algorithm. Machine Learning, 1997, 28(23): 133-168.

Krogh A, Vedelsby J. Neural network ensembles, cross validation, and active learning. In Proc. the 8th Annual Conference on Neural Information Processing Systems, November 1995, pp.231-238.

Burbidge R, Rowland J J, King R D. Active learning for regression based on query by committee. In Proc. the 8th Int. Conference on Intelligent Data Engineering and Automated Learning, December 2007, pp.209-218.

Cai W, Zhang Y, Zhou J. Maximizing expected model change for active learning in regression. In Proc. the 13th International Conference on Data Mining, December 2013, pp.51-60.

Bottou L. Large-scale machine learning with stochastic gradient descent. In Proc. the 19th Int. Conference on Computational Statistics, August 2010, pp.177-186.

Cai W, Zhang Y, Zhou S Y et al. Active learning for support vector machines with maximum model change. In Proc. the 2014 European Conference on Machine Learning and Knowledge Discovery in Databases, September 2014, pp.211-226.

Dasgupta S. The two faces of active learning. In Proc. the 20th Int. Conference on Algorithmic Learning Theory, October 2009, Article No. 1.

Dasgupta S, Hsu D. Hierarchical sampling for active learning. In Proc. the 25th Int. Conference on Machine Learning, June 2008, pp.208-215.

Urner R, Ben-David S. Probabilistic lipschitzness: A niceness assumption for deterministic labels. In Proc. the 27th NIPS Learning Faster from Easy Data Workshop, December 2013.

Steinwart I, Scovel C. Fast rates for support vector machines using Gaussian kernels. The Annals of Statistics, 2007, 35(2): 575-607.

Urner R, Shalev-Shwartz S, Ben-David S. Access to unlabeled data can speed up prediction time. In Proc. the 28th Int. Conference on Machine Learning, June 2011, pp.641-648.

Verma N, Kpotufe S, Dasgupta S. Which spatial partition trees are adaptive to intrinsic dimension? In Proc. the 25th Conference Uncertainty Artif. Intell., June 2009, pp.565-574.

Urner R, Wulff S, Ben-David S. PlAL: Cluster-based active learning. In Proc. the 26th Conference on Learning Theory, June 2013, pp.376-397.

Wang M, Min F, Zhang Z H, Wu Y X. Active learning through density clustering. Expert Systems with Applications, 2017, 85: 305-317.

Rodriguez A, Laio A. Clustering by fast search and find of density peaks. Science, 2014, 344(6191): 1492-1496.

Yan Y, Rosales R, Fung G, Dy J. Active learning from crowds. In Proc. the 28th Int. Conference on Machine Learning, June 2011, pp.1161-1168.

Fang M, Zhu X, Li B, Ding W, Wu X. Self-taught active learning from crowds. In Proc. the 12th Int. Conference on Data Mining, December 2012, pp.858-863.

Shu Z, Sheng V S, Li J. Learning from crowds with active learning and self-healing. Neural Computing and Applications, 2018, 30(9): 2883-2894.

Lampert C H, Nickisch H, Harmeling S. Attribute-based classification for zero-shot visual object categorization. IEEE Trans. Pattern Analysis and Machine Intelligence, 2014, 36(3): 453-465.

Ertekin S, Huang J, Giles C L. Active learning for class imbalance problem. In Proc. the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, July 2007, pp.823-824.

Attenberg J, Ertekin S¸. Class imbalance and active learning. In Imbalanced Learning: Foundations, Algorithms, and Applications, He H B, Ma Y Q (eds.), John Wiley & Sons, Inc., 2013, pp.101-149.

Tomanek K, Morik K. Inspecting sample reusability for active learning. In Proc. the Workshop on Active Learning and Experimental Design, May 2010, pp.169-181.

Hu R, Namee B M, Delany S J. Active learning for text classification with reusability. Expert Systems with Applications, 2016, 45(C): 438-449.

Settles B, Craven M, Friedland L. Active learning with real annotation costs. In Proc. the 2008 NIPS Workshop on Cost-Sensitive Learning, December 2008.

Tomanek K, Hahn U. A comparison of models for costsensitive active learning. In Proc. the 23rd Int. Conference on Computational Linguistics, August 2010, pp.1247-1255.

Liu A, Jun G, Ghosh J. Active learning of hyperspectral data with spatially dependent label acquisition costs. In Proc. the 2009 IEEE International Geoscience and Remote Sensing Symposium, July 2009, pp.256-259.

Persello C, Boularias A, Dalponte M et al. Cost-sensitive active learning with lookahead: Optimizing field surveys for remote sensing data classification. IEEE Trans. Geoscience and Remote Sensing, 2014, 52(10): 6652-6664.

Margineantu D. Active cost-sensitive learning. In Proc. the 19th International Joint Conference on Artificial Intelligence, July 2005, pp.1622-1623.

Krishnamurthy A, Agarwal A, Huang T et al. Active learning for cost-sensitive classification. arXiv:1703.01014, 2017. https://arxiv.org/abs/1703.01014, May 2019.

Zhang D, Wang F, Shi Z et al. Interactive localized content based image retrieval with multiple-instance active learning. Pattern Recognition, 2010, 43(2): 478-484.

Wang R, Wang X, Kwong S et al. Incorporating diversity and informativeness in multiple-instance active learning. IEEE Trans. Fuzzy Systems, 2017, 25(6): 1460-1475.

Wu J, Sheng V S, Zhang J, Zhao P, Cui Z. Multi-label active learning for image classification. In Proc. the 21st IEEE Int. Conference on Image Processing, October 2014, pp.5227-5231.

Yang B, Sun J T, Wang T, Chen Z. Effective multi-label active learning for text classification. In Proc. the 15th ACM SIGKDD Int. Conference on Knowledge Discovery and Data Mining, June 2009, pp.917-926.

Pupo O, Morell C, Ventura S. Effective active learning strategy for multi-label learning. Neurocomputing, 2017, 273: 494-508.

Cherman E A, Papanikolaou Y, Tsoumakas G et al. Multilabel active learning: Key issues and a novel query strategy. Evolving Systems, 2017, 10(1): 63-78.

Rani M, Dhok S, Deshmukh R. A systematic review of compressive sensing: Concepts, implementations and applications. IEEE Access, 2018, 6: 4875-4894.

Som S. Learning label structure for compressed sensing based multilabel classification. In Proc. the 2016 SAI Computing Conference, July 2016, pp.54-60.

Wu J, Ye C, Sheng V et al. Active learning with label correlation exploration for multi-label image classification. IET Computer Vision, 2017, 11(7): 577-584.

Pupo O, Ventural S. Evolutionary strategy to perform batch-mode active learning on multi-label data. ACM Trans. Intelligent Systems and Technology, 2018, 9(4): Article No. 46.

Reichart R, Tomanek K, Hahn U, Rappoport A. Multitask active learning for linguistic annotations. In Proc. the 46th Association for Computational Linguistics, June 2008, pp.861-869.

Zhang Y. Multi-task active learning with output constraints. In Proc. the 24th AAAI Conference on Artificial Intelligence, July 2010, pp.667-672.

Harpale A. Multi-task active learning [Ph.D. Thesis]. School of Computer Science, Carnegie Mellon University, 2012.

Gavves E, Mensink T, Tommasi T et al. Active transfer learning with zero-shot priors: Reusing past datasets for future tasks. In Proc. the 2015 IEEE International Conference on Computer Vision, December 2015, pp.2731-2739.

Wang X, Huang T, Schneider J. Active transfer learning under model shift. In Proc. the 31st Int. Conference on Machine Learning, June 2014, pp.1305-1313.

Guo Y, Schuurmans D. Discriminative batch mode active learning. In Proc. the 21st Annual Conference on Neural Information Processing Systems, December 2007, pp.593-600.

Chakraborty S, Balasubramanian V, Panchanathan S. Adaptive batch mode active learning. IEEE Trans. Neural Networks and Learning Systems, 2015, 26(8): 1747-1760.

Shen P, Li C, Zhang Z. Distributed active learning. IEEE Access, 2016, 4: 2572-2579.

Hinton G E, Osindero S, The Y. A fast learning algorithm for deep belief nets. Neural Computing, 2006, 18(7): 1527-1554.

Wang K, Zhang D, Li Y et al. Cost-effective active learning for deep image classification. IEEE Trans. Circuits and Systems for Video Technology, 2017, 27(12): 2591-2600.

Rahhal M M A, Bazi Y, Alhichri H et al. Deep learning approach for active classification of electrocardiogram signals. Information Sciences, 2016, 345(C): 340-354.

Zhou S, Chen Q, Wang X. Active deep learning method for semi-supervised sentiment classification. Neurocomputing, 2013, 120: 536-546.

Valiant L G. A theory of the learnable. Communications of the ACM, 1984, 27(11): 1134-1142.

Hanneke S. A bound on the label complexity of agnostic active learning. In Proc. the 24th Int. Conference on Machine Learning, June 2007, pp.353-360.

Hanneke S. Theoretical foundations of active learning [Ph.D. Thesis]. Machine Learning Department, CMU, 2009.

Hanneke S. Theory of disagreement-based active learning. Foundations and Trends in Machine Learning, 2014, 7(2/3): 131-309.

Dasgupta S. Coarse sample complexity bounds for active learning. In Proc. the 19th Annual Conference on Neural Information Processing Systems, December 2005, pp.235-242.

Tosh C, Dasgupta S. Diameter-based active learning. In Proc. the 34th International Conference on Machine Learning, August 2017, pp.3444-3452.

Audibert J Y, Tsybakov A B. Fast learning rates for plug-in classifiers. The Annals of Statistics, 2005, 35(2): 608-633.

Minsker S. Plug-in approach to active learning. Journal of Machine Learning Research, 2012, 13: 67-90.

Locatelli A, Carpentier A, Kpotufe S. Adaptivity to noise parameters in nonparametric active learning. In Proc. the 30th Conference on Learning Theory, July 2017, pp.1383-1416.

Schein A I, Ungar L H. Active learning for logistic regression: An evaluation. Machine Learning, 2007, 68(3): 235-265.

Melville P, Mooney R. Diverse ensembles for active learning. In Proc. the 21st Int. Conference on Machine Learning, July 2004, pp.584-591.

Yang Y, Loog M. A benchmark and comparison of active learning for logistic regression. Pattern Recognition, 2018, 83: 401-415.

Ramirez-Loaiza M E, Sharma M, Kumar G et al. Active learning: An empirical study of common baselines. Data Mining and Knowledge Discovery, 2017, 31(2): 287-313.

Pupo O, Altalhi H, Ventura S. Statistical comparisons of active learning strategies over multiple datasets. Knowledge-Based Systems, 2018, 145(1):274-288.

Merz C, Murphy P. UCI repository of machine learning databases. http://www.ics.uci.edu/ mlearn/MLRepository. html, Nov. 2019.

Frey PW, Slate D J. Letter recognition using Holland-style adaptive classifiers. Machine Learning, 1991, 6(2): 161-182.

Xu L, Krzyzak A, Suen C. Methods of combining multiple classifiers and their applications to handwritten recognition. IEEE Trans. Systems Man and Cybernetics, 1992, 22(3): 418-435.

Garofolo J, Lamel L, FisherWet al. DARPA TIMIT acoustic phonetic continuous speech corpus CD-ROM. Technical Report, 1993. https://nvlpubs.nist.gov/nistpubs/Legacy/IR/nistir4930.pdf, Nov. 2019.

Craven M, DiPasquo D, Freitag D et al. Learning to construct knowledge bases from theWorldWideWeb. Artificial Intelligence, 2000, 118(1/2): 69-113.

LeCun Y, Bottou L, Bengio Y et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998, 86(11): 2278-2324.

Lang K. NewsWeeder: Learning to filter net news. In Proc. the 12th Int. Conference on Machine Learning, July 1995, pp.331-339.

Deng J, Dong W, Socher R et al. ImageNet: A large-scale hierarchical image database. In Proc. the 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June 2009, pp.248-255.

Sang E F, de Meulder F. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proc. the 7th Conference on Natural Language Learning, May 2003, pp.142-147.

Collier N, Kim J. Introduction to the bio-entity recognition task at JNLPBA. In Proc. the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications, August 2004, Article No. 13.

Yeh A, Morgan A, Colosimo M et al. BioCreAtIvE task 1A: Gene mention finding evaluation. BMC Bioinformatics, 2005, 6(S-1): Article No. 2.

Vlachos A. Evaluating and combining biomedical named entity recognition systems. In Proc. the Workshop on Biological, Translational, and Clinical Language Processing, June 2007, pp.199-200.

Peng F, McCallum A. Information extraction from research papers using conditional random fields. Information Processing and Management, 2006, 42(4): 963-979

de Carvalho V R, Cohen W. Learning to extract signature and reply lines from email. In Proc. the 1st Conference on Email and Anti-Spam, July 2004.

Guyon I, Cawley G, Dror G et al. Results of the active learning challenge. In Proc. the Active Learning and Experimental Design Workshop, May 2010, pp.19-45.

Pace R K, Barry R. Sparse spatial autoregressions. Stat. Probab. Lett., 1997, 33(3): 291-297.

Bay S D, Kibler D, Pazzani M et al. The UCI KDD archive of large data sets for data mining research and experimentation. SIGKDD Explor., 2000, 2(2): 81-85.

Tang Y P, Li G X, Huang S J. ALiPy: Active learning in Python. arXiv:1901.03802, 2019. https://arxiv.org/abs/1901.03802, Nov. 2019.

Yang Y Y, Lee S C, Chung Y A et al. libact: Poolbased active learning in Python. arXiv:1710.00379, 2017. https://arxiv.org/abs/1710.00379, October 2019.

Tran V C, Nguyen N T, Fujita H et al. A combination of active learning and self-learning for named entity recognition on Twitter using conditional random fields. Knowledge-Based Systems, 2017, 132: 179-187.

Scheffer T, Decomain C, Wrobel S. Active hidden Markov models for information extraction. In Proc. the 4th Int. Conference on Advances in Intelligent Data Analysis, September 2001, pp.309-318.

Aldogan D, Yaslan Y. A comparison study on active learning integrated ensemble approaches in sentiment analysis. Computers and Electrical Engineering, 2017, 57(C): 311-323.

Zhang H, Huang M, Zhu X. A unified active learning framework for biomedical relation extraction. Journal of Computer Science and Technology, 2012, 27(6): 1302-1313.

Hoi S C H, Jin R, Zhu J et al. Batch mode active learning and its application to medical image classification. In Proc. the 23rd Int. Conference on Machine Learning, June 2006, pp.417-424.

Wallace B C, Small K, Brodley C et al. Active learning for biomedical citation screening. In Proc. the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, July 2010, pp.173-182.

Ma A, Patel N, Li M et al. Confidence based active learning for whole object image segmentation. In Proc. the 2006 Int. Workshop on Multimedia Content Representation, Classification and Security, September 2006, pp.753-760.

Pavlopoulou, Kak A, Brodley C. Application of semisupervised and active learning to interactive contour delineation. In Proc. the ICML 2003 Workshop on the Continuum from Labeled to Unlabeled Data in Machine Learning and Data Mining, August 2003, pp.26-33.

Boutell M R, Luo J, Shen X et al. Learning multi-label scene classification. Pattern Recognition, 2004, 37(9): 1757-1771.

Zhang B, Wang Y, Chen F. Multilabel image classification via high-order label correlation driven active learning. IEEE Trans. Image Processing, 2014, 23(3): 1430-1441.

Top A, Hamarneh G, Abugharbieh R. Active learning for interactive 3D image segmentation. In Proc. the 14th Int. Conference on Medical Image Computing and Computerassisted Intervention, September 2011, pp.603-610.

Caicedo J C, Lazebnik S. Active object localization with deep reinforcement learning. In Proc. the 2015 IEEE Int. Conference on Computer Vision, December 2015, pp.2488-2496.

Kim Y, Kim S. Design of aging-resistant Wi-Fi fingerprintbased localization system with continuous active learning. In Proc. the 20th Int. Conference on Advanced Communication Technology, February 2018, pp.s1054-1059.

Ayache S, Qu´enot G. Video corpus annotation using active learning. In Proc. the 30th European Conference on Information Retrieval Research, March 2008, pp.187-198.

Reker D, Schneider G. Active-learning strategies in computer-assisted drug discovery. Drug Discovery Today, 2015, 20(4): 458-465.

Warmuth M K, Rätsch G, Mathieson M et al. Active learning in the drug discovery process. In Proc. the 15th Annual Conference on Neural Information Processing Systems, December 2001, pp.1449-1456.

Figueroa R L, Zeng-Treitler Q, Ngo L et al. Active learning for clinical text classification: Is it better than random sampling? Journal of the American Medical Informatics Association, 2012, 19(5): 809-816.

Chen Y, Lasko T, Mei Q et al. A study of active learning methods for named entity recognition in clinical text. Journal of Biomedical Informatics, 2015, 58(1): 11-18.

Gu Y, Zydek D. Active learning for intrusion detection. In Proc. the 2014 National Wireless Research Collaboration Symposium, May 2014, pp.117-122.

Hossain H M S, Roy N, Khan M. Active learning enabled activity recognition. In Proc. the 2016 IEEE Int. Conference on Pervasive Computing and Communications, March 2016, Article No. 26.

Reker D, Schneider P, Schneider G. Multi-objective active machine learning rapidly improves structure-activity models and reveals new protein-protein interaction inhibitors. Chemical Science, 2016, 7(6): 3919-3927.

Yan S, Chaudhuri K, Javidi T. Active learning with logged data. arXiv:1802.09069, 2018. https://arxiv.org/abs/1802.09069, Nov. 2019.

Danka T, Horvath P. modAL: A modular active learning framework for Python. arXiv:1805.00979, 2018. https://arxiv.org/abs/1805.00979, Nov. 2019.

Pedregosa F, Varoquaux G, Gramfort A et al. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 2011, 12: 2825 -2830.

Atienza R. Advanced Deep Learning with Keras: Apply Deep Learning Techniques, Autoencoders, GANs, Variational Autoencoders, Deep Reinforcement Learning, Policy Gradients, and More. Packt Publishing, 2018.