Human-in-the-loop machine learning: a state of the art

Abdul A, Vermeulen J, Wang D et al (2018) Trends and trajectories for explainable, accountable and intelligible systems: an hci research agenda. In: Proceedings of the 2018 CHI conference on human factors in computing systems. Association for Computing Machinery, New York, NY, USA, CHI ’18, pp 1–18, https://doi.org/10.1145/3173574.3174156 Abiteboul S, Buneman P, Suciu D (2000) Data on the web: from relations to semistructured data and XML. Morgan Kaufmann, Data Management Systems Series Adadi A, Berrada M (2018) Peeking inside the black-box: a survey on explainable artificial intelligence (XAI). IEEE Access 6:52,138-52,160. https://doi.org/10.1109/ACCESS.2018.2870052 Aggarwal CC, Kong X, Gu Q et al (2014) Active learning: a survey. Data classification: algorithms and applications. Chapman and Hall/CRC, Boca Raton, pp 599–634 Amazon (2022) Amazon mechanical turk. https://www.mturk.com/. Accessed on 23 Mar 2022 Amershi S, Cakmak M, Knox WB et al (2014) Power to the people: the role of humans in interactive machine learning. AI Magazine 35(4):105–120. https://doi.org/10.1609/aimag.v35i4.2513 Angluin D (1987) Learning regular sets from queries and counterexamples. Inf Comput 75(2):87–106. https://doi.org/10.1016/0890-5401(87)90052-6 Arras GL, Montavon, Müller KR, Samek W (2017) Explaining recurrent neural network predictions in sentiment analysis. In: EMNLP’17 workshop on computational approaches to subjectivity, sentiment and social media analysis, https://doi.org/10.18653/v1/W17-5221 Barakat NH, Bradley AP (2007) Rule extraction from support vector machines: a sequential covering approach. IEEE Trans Knowl Data Eng 19(6):729–741. https://doi.org/10.1109/TKDE.2007.190610 Barredo Arrieta A, Díaz-Rodríguez N, Del Ser J et al (2020) Explainable artificial intelligence (xai): concepts, taxonomies, opportunities and challenges toward responsible AI. Inf Fusion 58:82–115. https://doi.org/10.1016/j.inffus.2019.12.012 Bau D, Zhou B, Khosla A et al (2017) Network dissection: Quantifying interpretability of deep visual representations. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), 3319–3327, https://doi.org/10.1109/CVPR.2017.354, https://ieeexplore.ieee.org/document/8099837 Begeja L, Renger B, Gibbon D et al (2004) Interactive machine learning techniques for improving SLU models. In: Proceedings of the HLT-NAACL 2004 workshop on spoken language understanding for conversational systems and higher level linguistic information for speech processing. Association for Computational Linguistics, Boston, Massachusetts, USA, 10–16, https://aclanthology.org/W04-3003 Bengio Y, Louradour J, Collobert R et al (2009) Curriculum learning. In: Proceedings of the 26th annual international conference on machine learning. Association for Computing Machinery, New York, NY, USA, ICML ’09, 41-48, https://doi.org/10.1145/1553374.1553380 Bennetot A, Laurent JL, Chatila R et al (2019) Towards explainable neural-symbolic visual reasoning. arxiv:1909.09065 Berg S, Kutra D, Kroeger T et al (2019) Ilastik: interactive machine learning for (bio)image analysis. Nat Methods 16(12):1226–1232. https://doi.org/10.1038/s41592-019-0582-9 Berghel H (1997) Cyberspace 2000: dealing with information overload. Commun ACM 40(2):19–24. https://doi.org/10.1145/253671.253680 Blumberg R, Atre S (2003) The problem with unstructured data. DM Rev 13(42–49):62 Bonwell CC, Eison JA (1991) Active learning: creating excitement in the classroom. 1991 ASHE-ERIC higher education reports. ERIC Clearinghouse on Higher Education, The George Washington University, One Dupont Circle, Suite 630, Washington, DC 20036-1183 Boukhelifa N, Bezerianos A, Lutton E (2018) Evaluation of interactive machine learning systems. In: Zhou J, Chen F (eds) Human and machine learning: visible, explainable, trustworthy and transparent. Springer, Cham, pp 341–360 Budd S, Robinson EC, Kainz B (2021) A survey on active learning and human-in-the-loop deep learning for medical image analysis. Med Image Anal 71(102):062. https://doi.org/10.1016/j.media.2021.102062 Carlson G (2015) What eactly is complex data? https://www.ayasdi.com/exactly-complex-data/. Accessed on 04 Mar 2021 Castle E (2017) 7 signs you’re dealing with complex data. https://www.sisense.com/blog/7-signs-youre-dealing-with-complex-data/. Accessed on 04 Mar 2022 Che Z, Purushotham S, Khemani R et al (2015) Distilling knowledge from deep networks with applications to healthcare domain. arXiv e-prints arxiv:1512.03542 [stat.ML] Che Z, Purushotham S, Khemani R et al (2017) Interpretable deep models for ICU outcome prediction. In: AMIA annual symposium proceedings, 371–380, https://pubmed.ncbi.nlm.nih.gov/28269832/ Chen Z, Li J, Wei L (2007) A multiple kernel support vector machine scheme for feature selection and rule extraction from gene expression data of cancer tissue. Artif Intell Med 41(2):161–175. https://doi.org/10.1016/j.artmed.2007.07.008 Chen Y, Singla A, Aodha OM et al (2018) Understanding the role of adaptivity in machine teaching: The case of version space learners. In: Proceedings of the 32nd international conference on neural information processing systems. Curran Associates Inc., Red Hook, NY, USA, NIPS’18, 1483-1493, https://dl.acm.org/doi/abs/10.5555/3326943.3327079 Choi E, Bahadori T, Schuetz A et al (2016) Retain: Interpretable predictive model in healthcare using reverse time attention mechanism. In: Proceedings of the 30th international conference on neural information processing systems. Curran Associates Inc., Red Hook, NY, USA, NIPS’16, 3512-3520 Churchill EF, van Allen P, Kuniavsky M (2018) Designing AI. Interactions 25(6):34–37. https://doi.org/10.1145/3281764 Cirik V, Hovy E, Morency LP (2016) Visualizing and understanding curriculum learning for long short-term memory networks. arXiv e-prints arxiv:1611.06204 [cs.CL] Cohn D, Atlas L, Ladner R (1994) Improving generalization with active learning. Mach Learn 15(2):201–221. https://doi.org/10.1007/BF00993277 Cohn DA, Ghahramani Z, Jordan MI (1996) Active learning with statistical models. J Artif Intell Res 4(1):129–145. https://doi.org/10.5555/1622737.1622744 d’Avila Garcez A, Gori M, Lamb LC et al (2019) Neural-symbolic computing: an effective methodology for principled integration of machine learning and reasoning. arXiv e-prints arxiv:1905.06088 [cs.AI] Das A, Rad P (2020) Opportunities and challenges in explainable artificial intelligence (XAI): a survey. arXiv e-prints arxiv:2006.11371 [cs.CV] De Angeli K, Gao S, Alawad M et al (2021) Deep active learning for classifying cancer pathology reports. BMC Bioinform 22(1):1–25 Devidze R, Mansouri F, Haug L et al (2020) Understanding the power and limitations of teaching with imperfect knowledge. In: Bessiere C (ed) Proceedings of the twenty-ninth international joint conference on artificial intelligence, IJCAI-20. International Joint Conferences on Artificial Intelligence Organization, 2647–2654, https://doi.org/10.24963/ijcai.2020/367 Diamant E (2009) Machine learning: When and where the horses went astray? In: Zhang Y (ed) Machine learning. InTech, London, pp 1–18. https://doi.org/10.5772/9156 Diamant E (2006) Learning to understand image content: Machine learning versus machine teaching alternative. In: 2006 International conference on information technology: research and education, 26–29, https://doi.org/10.1109/ITRE.2006.381526 Donadello I, Kessler F, Dragoni M et al (2019) Persuasive explanation of reasoning inferences on dietary data. In: Joint proceedings of the 6th international workshop on dataset profilling and search and the 1st workshop on semantic explainability co-located with the 18th international semantic web conference (ISWC 2019) Donmez P, Carbonell JG (2008) Proactive learning: Cost-sensitive active learning with multiple imperfect oracles. In: Proceedings of the 17th ACM conference on information and knowledge management. Association for Computing Machinery, New York, NY, USA, CIKM ’08, 619–628, https://doi.org/10.1145/1458082.1458165 Dudley JJ, Kristensson PO (2018) A review of user interface design for interactive machine learning. ACM Trans Interact Intell Syst. https://doi.org/10.1145/3185517 El-Hasnony IM, Elzeki OM, Alshehri A et al (2022) Multi-label active learning-based machine learning model for heart disease prediction. Sensors. https://doi.org/10.3390/s22031184 Elman JL (1993) Learning and development in neural networks: the importance of starting small. Cognition 48(1):71–99. https://doi.org/10.1016/0010-0277(93)90058-4 Fadhil A, Wang Y (2018) Towards automatic & personalised mobile health interventions: an interactive machine learning perspective. arXiv e-prints arxiv:1803.01842 [cs.CY] Fails JA, Olsen DR (2003) Interactive machine learning. In: Proceedings of the 8th international conference on intelligent user interfaces. Association for Computing Machinery, New York, NY, USA, IUI ’03, 39–45, https://doi.org/10.1145/604045.604056 Fiebrink RA (2011) Real-time human interaction with supervised learning algorithms for music composition and performance. PhD thesis, Computer Science Dept. Princeton University, Princeton, NJ, USA, https://dl.acm.org/doi/book/10.5555/2125776 Fiebrink R, Cook PR (2010) The wekinator: a system for real-time, interactive machine learning in music. In: Proceedings of The Eleventh International Society for Music Information Retrieval Conference (ISMIR 2010), Utrecht Fiebrink R, Cook PR, Trueman D (2011) Human model evaluation in interactive supervised learning. In: Proceedings of the SIGCHI conference on human factors in computing systems. Association for Computing Machinery, New York, NY, USA, CHI ’11, 147–156, https://doi.org/10.1145/1978942.1978965 Florensa C, Held D, Wulfmeier M et al (2017) Reverse curriculum generation for reinforcement learning. In: Levine S, Vanhoucke V, Goldberg K (eds) Proceedings of the 1st annual conference on robot learning, proceedings of machine learning research, vol 78. PMLR, 482–495, http://proceedings.mlr.press/v78/florensa17a.html Fogarty J, Tan D, Kapoor A et al (2008) Cueflik: Interactive concept learning in image search. In: Proceedings of the SIGCHI conference on human factors in computing systems. Association for Computing Machinery, New York, NY, USA, CHI ’08, 29–38, https://doi.org/10.1145/1357054.1357061 Gaonkar B, Shinohara TR, Davatzikos C (2015) Interpreting support vector machine models for multivariate group wise analysis in neuroimaging. Med Image Anal 24(1):190–204. https://doi.org/10.1016/j.media.2015.06.008 Goodman B, Flaxman S (2017) European union regulations on algorithmic decision-making and a “right to explanation.” AI Mag 38(3):50–57. https://doi.org/10.1609/aimag.v38i3.2741 Gunning D (2017) Explainable artificial intelligence (xAI). Tech. rep., Defense Advanced Research Projects Agency (DARPA), https://www.darpa.mil/program/explainable-artificial-intelligence Hacohen G, Weinshall D (2019) On the power of curriculum learning in training deep networks. In: Chaudhuri K, Salakhutdinov R (eds) Proceedings of the 36th international conference on machine learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, Proceedings of Machine Learning Research, vol 97. PMLR, 2535–2544, http://proceedings.mlr.press/v97/hacohen19a.html Hara S, Hayashi K (2018) Making tree ensembles interpretable: A bayesian model selection approach. In: Storkey A, Perez-Cruz F (eds) Proceedings of the twenty-first international conference on artificial intelligence and statistics, proceedings of machine learning research, vol 84. PMLR, 77–85, https://proceedings.mlr.press/v84/hara18a.html He X, Zhao K, Chu X (2021) AutoML: a survey of the state-of-the-art. Knowl Based Syst 212(106):622. https://doi.org/10.1016/j.knosys.2020.106622 Heimerl F, Koch S, Bosch H et al (2012) Visual classifier training for text document retrieval. IEEE Trans Vis Comput Graphics 18(12):2839–2848. https://doi.org/10.1109/TVCG.2012.277 Hills TT, Todd PM, Lazer D et al (2015) Exploration versus exploitation in space, mind, and society. Trends Cogn Sci 19(1):46–54. https://doi.org/10.1016/j.tics.2014.10.004 Hipke K, Toomim M, Fiebrink R et al (2014) Beatbox: End-user interactive definition and training of recognizers for percussive vocalizations. In: Proceedings of the 2014 international working conference on advanced visual interfaces. Association for Computing Machinery, New York, NY, USA, AVI ’14, 121–124, https://doi.org/10.1145/2598153.2598189 Hoi SCH, Jin R, Zhu J et al (2006) Batch mode active learning and its application to medical image classification. In: Proceedings of the 23rd international conference on machine learning. Association for Computing Machinery, New York, NY, USA, ICML ’06, 417-424, https://doi.org/10.1145/1143844.1143897 Holmberg L, Davidsson P, Linde P (2020) A feature space focus in machine teaching. In: 2020 IEEE international conference on pervasive computing and communications workshops (PerCom Workshops), 1–2, https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 Holzinger A (2016) Interactive machine learning for health informatics: when do we need the human-in-the-loop? Brain Inform 3(2):119–131. https://doi.org/10.1007/s40708-016-0042-6 Holzinger A, Jurisica I (2014) Knowledge discovery and data mining in biomedical informatics: the future is in integrative, interactive machine learning solutions. In: Holzinger A, Jurisica I (eds) Interactive knowledge discovery and data mining in biomedical informatics: state-of-the-art and future challenges. Springer, Berlin, Heidelberg, pp 1–18. https://doi.org/10.1007/978-3-662-43968-5_1 Holzinger A, Plass M, Kickmeier-Rust M et al (2019) Interactive machine learning: experimental evidence for the human in the algorithmic loop. Appl Intell 49(7):2401–2414. https://doi.org/10.1007/s10489-018-1361-5 Holzinger A, Biemann C, Pattichis CS, et al (2017) What do we need to build explainable AI systems for the medical domain? arXiv e-prints arxiv:1712.09923 [cs.AI] Ionescu RT, Alexe B, Leordeanu M et al (2016) How hard can it be? estimating the difficulty of visual search in an image. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 2157–2166, https://doi.org/10.1109/CVPR.2016.237 Ishibashi T, Nakao Y, Sugano Y (2020) Investigating audio data visualization for interactive sound recognition. In: Proceedings of the 25th international conference on intelligent user interfaces. Association for Computing Machinery, New York, NY, USA, IUI ’20, 67–77, https://doi.org/10.1145/3377325.3377483 Jamieson KG, Jain L, Fernandez C et al (2015) Next: a system for real-world development, evaluation, and application of active learning. In: Cortes C, Lawrence N, Lee D et al (eds) Advances in neural information processing systems, vol 28. Curran Associates Inc, Red Hook Jiang L, Meng D, Zhao Q et al (2015) Self-paced curriculum learning. In: Proceedings of the twenty-ninth AAAI conference on artificial intelligence. AAAI Press, AAAI’15, 2694-2700, https://doi.org/10.5555/2886521.2886696 Jiang L, Liu S, Chen C (2019) Recent research advances on interactive machine learning. J Vis 22(2):401–417. https://doi.org/10.1007/s12650-018-0531-1 Johns E, Mac Aodha O, Brostow GJ (2015) Becoming the expert-interactive multi-class machine teaching. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2616–2624, https://doi.org/10.1109/CVPR.2015.7298877 Kabra M, Robie AA, Rivera-Alba M et al (2013) Jaaba: interactive machine learning for automatic annotation of animal behavior. Nat Methods 10(1):64–67. https://doi.org/10.1038/nmeth.2281 Kapoor A, Lee B, Tan D et al (2010) Interactive optimization for steering machine classification. In: Proceedings of the SIGCHI conference on human factors in computing systems. Association for Computing Machinery, New York, NY, USA, CHI ’10, 1343–1352, https://doi.org/10.1145/1753326.1753529 Kellenberger B, Tuia D, Morris D (2020) Aide: accelerating image-based ecological surveys with interactive machine learning. Methods Ecol Evol 11(12):1716–1727. https://doi.org/10.1111/2041-210X.13489 Kim B, Patel K, Rostamizadeh A et al (2015) Scalable and interpretable data representation for high-dimensional, complex data. In: Proceedings of the twenty-ninth AAAI conference on artificial intelligence, Association for the Advancement of Artificial Intelligence (AAAI), Austin, Texas USA, 1763–1769, https://ojs.aaai.org/index.php/AAAI/article/view/9474 Koesten L, Simperl E (2021) Ux of data: making data available doesn’t make it usable. Interactions 28(2):97–99. https://doi.org/10.1145/3448888 Kosmyna N, Tarpin-Bernard F, Rivet B (2015) Adding human learning in brain-computer interfaces (bcis): towards a practical control modality. ACM Trans Comput-Hum Interact. https://doi.org/10.1145/2723162 Krakovna V, Doshi-Velez F (2016) Increasing the interpretability of recurrent neural networks using hidden markov models. arXiv e-prints arxiv:1606.0532 [cond-mat.soft] Kumar M, Packer B, Koller D (2010) Self-paced learning for latent variable models. In: Lafferty J, Williams C, Shawe-Taylor J et al (eds) Advances in neural information processing systems. Curran Associates Inc, Red Hook, pp 1189–1197 Kumar G, Foster G, Cherry C et al (2019) Reinforcement learning based curriculum optimization for neural machine translation. In: Proceedings of the 2019 conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 2054–2061, https://doi.org/10.18653/v1/N19-1208, https://www.aclweb.org/anthology/N19-1208 Laws F, Scheible C, Schütze H (2011) Active Learning with Amazon Mechanical Turk. In: Proceedings of the conference on empirical methods in natural language processing. Association for Computational Linguistics, USA, EMNLP ’11, 1546-1556, https://doi.org/10.5555/2145432.2145597 LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444. https://doi.org/10.1038/nature14539 Leslie D (2019) Understanding artificial intelligence ethics and safety: A guide for the responsible design and implementation of ai systems in the public sector. 10.5281/zenodo.3240529 Lindvall M, Molin J, Löwgren J (2018) From machine learning to machine teaching: the importance of UX. Interactions 25(6):52–57. https://doi.org/10.1145/3282860 Lipton ZC (2018) The mythos of model interpretability: in machine learning, the concept of interpretability is both important and slippery. Queue 16(3):31–57. https://doi.org/10.1145/3236386.3241340 Liu W, Dai B, Humayun A et al (2017) Iterative machine teaching. In: Precup D, Teh YW (eds) Proceedings of the 34th international conference on machine learning, proceedings of machine learning research, vol 70. PMLR, 2149–2158, https://proceedings.mlr.press/v70/liu17b.html Liu C, He S, Liu K et al (2018a) Curriculum learning for natural answer generation. In: Proceedings of the 27th international joint conference on artificial intelligence. AAAI Press, IJCAI’18, 4223-4229, https://doi.org/10.24963/ijcai.2018/587 Liu J, Lichtenberg T, Hoadley KA et al (2018b) An integrated TCGA pan-cancer clinical data resource to drive high-quality survival outcome analytics. Cell 173(2):400-416.e11. https://doi.org/10.1016/j.cell.2018.02.052 Liu W, Dai B, Li X et al (2018c) Towards black-box iterative machine teaching. In: Dy J, Krause A (eds) Proceedings of the 35th international conference on machine learning, proceedings of machine learning research, vol 80. PMLR, 3141–3149, https://proceedings.mlr.press/v80/liu18b.html Liu Z, Feng X, Wang Y et al (2021) Self-paced learning enhanced neural matrix factorization for noise-aware recommendation. Knowl Based Syst 213(106):660. https://doi.org/10.1016/j.knosys.2020.106660 Lopes M, Melo F, Montesano L (2009) Active learning for reward estimation in inverse reinforcement learning. Joint European conference on machine learning and knowledge discovery in databases. Springer, Berlin Heidelberg, pp 31–46 Loyola-González O (2019) Black-box vs. white-box: understanding their advantages and weaknesses from a practical point of view. IEEE Access 7:154096–154113. https://doi.org/10.1109/ACCESS.2019.2949286 Luo T, Kramer K, Samson S et al (2004) Active learning to recognize multiple types of plankton. In: Proceedings of the 17th international conference on pattern recognition, 2004. ICPR 2004., 478–481 Vol.3, https://doi.org/10.1109/ICPR.2004.1334570 Mahendran A, Vedaldi A (2015) Understanding deep image representations by inverting them. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), 5188–5196, https://doi.org/10.1109/CVPR.2015.7299155, https://ieeexplore.ieee.org/document/7299155 Matiisen T, Oliver A, Cohen T et al (2020) Teacher-student curriculum learning. IEEE Trans Neural Netw Learn Syst 31(9):3732–3740. https://doi.org/10.1109/TNNLS.2019.2934906 Mei S, Zhu X (2015) Using machine teaching to identify optimal training-set attacks on machine learners. In: Proc. of the 29th AAAI conference on artificial intelligence, 2871–2877, https://ojs.aaai.org/index.php/AAAI/article/view/9569 Meske C, Bunde E, Schneider J et al (2022) Explainable artificial intelligence: objectives, stakeholders, and future research opportunities. Inf Syst Manag 39(1):53–63. https://doi.org/10.1080/10580530.2020.1849465 Meza Martínez MA, Nadj M, Maedche A (2019) Towards an integrative theoretical framework of interactive machine learning systems. In: Proceedings of the 27th European conference on information systems (ECIS), Stockholm & Uppsala, Sweden, https://aisel.aisnet.org/ecis2019_rp/172 Michael CJ, Acklin D, Scheuerman J (2020) On interactive machine learning and the potential of cognitive feedback. arXiv e-prints arxiv:2003.10365 [cs.HC] Microsoft (2022) Qna maker. https://www.qnamaker.ai/. Accessed on 23 Mar 2022 Minh D, Wang HX, Li YF et al (2021) Explainable artificial intelligence: a comprehensive review. Artif Intell Rev. https://doi.org/10.1007/s10462-021-10088-y Montavon G, Lapuschkin S, Binder A et al (2017) Explaining nonlinear classification decisions with deep Taylor decomposition. Pattern Recognit 65:211–222. https://doi.org/10.1016/j.patcog.2016.11.008 Mosqueira-Rey E, Alonso-Ríos D, Baamonde-Lozano A (2021) Integrating iterative machine teaching and active learning into the machine learning loop. Procedia Comput Sci 192:553–562. https://doi.org/10.1016/j.procs.2021.08.057 Mosqueira-Rey E, Hernández-Pereira E, Alonso-Ríos D et al (2022) A classification and review of tools for developing and interacting with machine learning systems. In: Proceedings of the 37th annual ACM symposium on applied computing. Association for Computing Machinery, New York, NY, USA, 1083–1092, https://doi.org/10.1145/3477314.3507310 Muir BM (1987) Trust between humans and machines, and the design of decision aids. Int J Man-Mach Stud 27(5):527–539. https://doi.org/10.1016/S0020-7373(87)80013-5 Munro R (2020) Human-in-the-loop machine learning. Manning Publications, Shelter Island Nguyen DHM, Patrick JD (2014) Supervised machine learning and active learning in classification of radiology reports. J Am Med Inform Assoc 21(5):893–901. https://doi.org/10.1136/amiajnl-2013-002516 Nguyen A, Dosovitskiy A, Yosinski J et al (2016) Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In: Proceedings of the 30th international conference on neural information processing systems. Curran Associates Inc., Red Hook, NY, USA, NIPS’16, 3395-3403, https://doi.org/10.5555/3157382.3157477 Nwana HS (1990) Intelligent tutoring systems: an overview. Artif Intell Rev 4(4):251–277. https://doi.org/10.1007/BF00168958 Olsson F (2009) A literature survey of active machine learning in the context of natural language processing. Tech. rep., Swedish Institute of Computer Science, http://urn.kb.se/resolve?urn=urn:nbn:se:ri:diva-23510 O’Malley J (2018) Captcha if you can: how you’ve been training ai for years without realising it. https://www.techradar.com/news/captcha-if-you-can-how-youve-been-training-ai-for-years-without-realising-it Peng B, Li C, Li J et al (2021) Soloist: building task bots at scale with transfer learning and machine teaching. Trans Assoc Comput Linguist 9:807–824. https://doi.org/10.1162/tacl_a_00399 Penha G, Hauff C (2020) Curriculum learning strategies for IR. In: Jose JM, Yilmaz E, Magalhães J et al (eds) European conference on information retrieval: advances in information retrieval. Springer, Cham, pp 699–713. https://doi.org/10.1007/978-3-030-45439-5_46 Platanios EA, Stretcu O, Neubig G et al (2019) Competence-based curriculum learning for neural machine translation. In: Proceedings of the 2019 conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, pp 1162–1172, https://doi.org/10.18653/v1/N19-1119, https://www.aclweb.org/anthology/N19-1119 Porter R, Theiler J, Hush D (2013) Interactive machine learning in data exploitation. Comput Sci Eng 15(5):12–20. https://doi.org/10.1109/MCSE.2013.74 Ramos G, Meek C, Simard P et al (2020) Interactive machine teaching: a human-centered approach to building machine-learned models. Hum Comput Interact 35(5–6):413–451. https://doi.org/10.1080/07370024.2020.1734931 Reyes O, Pérez E, del Carmen Rodrıguez-Hernández M et al (2016) Jclal: a java framework for active learning. J Mach Learn Res 17:1–5 Ribeiro M, Grolinger K, Capretz MA (2015) MLaaS: Machine learning as a service. In: 2015 IEEE 14th international conference on machine learning and applications (ICMLA), 896–902, https://doi.org/10.1109/ICMLA.2015.152 Rubens N, Elahi M, Sugiyama M et al (2015) Active learning in recommender systems. In: Ricci F, Rokach L, Shapira B (eds) Recommender systems handbook. Springer, Boston, pp 809–846 Rusu O, Halcu I, Grigoriu O et al (2013) Converting unstructured and semi-structured data into knowledge. In: 013 11th RoEduNet international conference, 1–4, https://doi.org/10.1109/RoEduNet.2013.6511736 Sammut C, Banerji RB (1986) Learning concepts by asking questions. In: Michalski RS, Carbonell J, Mitchell T (eds) Machine learning: an artificial intelligence approach, vol 2. Morgan Kaufmann, Burlington, pp 167–192 Šavelka J, Trivedi G, Ashley KD (2015) Applying an interactive machine learning approach to statutory analysis. In: Rotolo A (ed) Legal knowledge and information systems, frontiers in artificial intelligence and applications, vol 279. IOS Press, Amsterdam, pp 101–110. https://doi.org/10.3233/978-1-61499-609-5-101 Selvaraju RR, Cogswell M, Das A et al (2017) Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE international conference on computer vision (ICCV), 618–626, https://doi.org/10.1109/ICCV.2017.74, https://ieeexplore.ieee.org/document/8237336 Sena A, Howard M (2020) Quantifying teaching behavior in robot learning from demonstration. Int J Robot Res 39(1):54–72. https://doi.org/10.1177/0278364919884623 Sena A, Zhao Y, Howard MJ (2018) Teaching human teachers to teach robot learners. In: 2018 IEEE international conference on robotics and automation (ICRA), 5675–5681, https://doi.org/10.1109/ICRA.2018.8461194 Settles B (2009) Active learning literature survey. Tech. rep., University of Wisconsin-Madison. Department of Computer Sciences, https://minds.wisconsin.edu/handle/1793/60660 Settles B (2011) From theories to queries: Active learning in practice. In: Guyon I, Cawley G, Dror G et al (eds) Active learning and experimental design workshop In conjunction with AISTATS 2010, proceedings of machine learning research, vol 16. JMLR workshop and conference proceedings, Sardinia, Italy, 1–18, http://proceedings.mlr.press/v16/settles11a.html Shneiderman B (2020) Human-centered artificial intelligence: reliable, safe & trustworthy. Int J Hum Comput Interact 36(6):495–504. https://doi.org/10.1080/10447318.2020.1741118 Simard PY, Amershi S, Chickering DM et al (2017) Machine teaching: A new paradigm for building machine learning systems. arXiv e-prints arxiv:1707.06742 Singla A, Bogunovic I, Bartók G et al (2014) Near-optimally teaching the crowd to classify. In: Xing EP, Jebara T (eds) Proceedings of the 31st international conference on machine learning. PMLR, Bejing, China, proceedings of machine learning research, 154–162, http://proceedings.mlr.press/v32/singla14.pdf Sint R, Schaffert S, Stroka S et al (2009) Combining unstructured, fully structured and semi-structured information in semantic wikis. In: 4th semantic wiki workshop (SemWiki 2009) at the 6th European semantic web conference (ESWC 2009), Hersonissos, Greece, 73–87, http://ceur-ws.org/Vol-464/paper-14.pdf Smith JS, Nebgen B, Lubbers N et al (2018) Less is more: sampling chemical space with active learning. J Chem Phys 148(24):241,733 Soviany P, Ardei C, Ionescu RT et al (2020) Image difficulty curriculum for generative adversarial networks (cugan). In: 2020 IEEE winter conference on applications of computer vision (WACV), 3452–3461, https://doi.org/10.1109/WACV45572.2020.9093408 Spitkovsky VI, Alshawi H, Jurafsky D (2010) From baby steps to leapfrog: How “less is more” in unsupervised dependency parsing. In: Human language technologies: the 2010 annual conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Los Angeles, California, 751–759, https://www.aclweb.org/anthology/N10-1116 Suh J, Ghorashi S, Ramos G et al (2019) Anchorviz: facilitating semantic data exploration and concept discovery for interactive machine learning. ACM Trans Interact Intell Syst. https://doi.org/10.1145/3241379 Talbot J, Lee B, Kapoor A et al (2009) Ensemblematrix: Interactive visualization to support machine learning with multiple classifiers. In: Proceedings of the SIGCHI conference on human factors in computing systems. Association for Computing Machinery, New York, NY, USA, CHI ’09, 1283-1292, https://doi.org/10.1145/1518701.1518895 Tang YP, Li GX, Huang SJ (2019) ALiPy: Active learning in python. Tech. rep., Nanjing University of Aeronautics and Astronautics, https://github.com/NUAA-AL/ALiPy, available as arXiv preprint arxiv:1901.03802 Teso S, Kersting K (2019) Explanatory interactive machine learning. In: Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. Association for Computing Machinery, New York, NY, USA, AIES ’19, 239–245, https://doi.org/10.1145/3306618.3314293 Tolls V (2018) An event-based approach to modeling complex data in critical care. PhD thesis, Queen’s University (Canada), https://qspace.library.queensu.ca/bitstream/handle/1974/24489/Tolls_Victoria_J_201809_MSC.pdf Tomczak K, Czerwińska P, Wiznerowicz M (2015) The cancer genome atlas (TCGA): an immeasurable source of knowledge. Contemp Oncol 19(1A):68–77. https://doi.org/10.5114/wo.2014.47136 Treveil M, Omont N, Stenac C et al (2020) Introducing MLOps. O’Reilly Media, Sebastopol Traoré R, Caselles-Dupré ea (2019) Discorl: continual reinforcement learning via policy distillation. arXiv e-prints arxiv:1907.05855 [cs.LG] Valiant LG (1984) A theory of the learnable. Commun ACM 27(11):1134–1142. https://doi.org/10.1145/1968.1972 van Allen P (2018) Prototyping ways of prototyping AI. Interactions 25(6):46–51. https://doi.org/10.1145/3274566 von Ahn L, Dabbish L (2004) Labeling images with a computer game. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, New York, NY, USA, CHI ’04, 319–326, https://doi.org/10.1145/985692.985733 von Ahn L, Blum M, Langford J (2004) Telling humans and computers apart automatically. Commun ACM 47(2):56–60. https://doi.org/10.1145/966389.966390 Visi FG, Tanaka A (2021) Interactive machine learning of musical gesture. In: Miranda ER (ed) Handbook of artificial intelligence for music: foundations, advanced approaches, and developments for creativity. Springer, Cham, pp 771–798. https://doi.org/10.1007/978-3-030-72116-9_27 Wall E, Ghorashi S, Ramos G (2019) Using expert patterns in assisted interactive machine learning: a study in machine teaching. In: Lamas D, Loizides F, Nacke L et al (eds) Human-computer interaction—INTERACT 2019. Springer, Berlin, pp 578–599. https://doi.org/10.1007/978-3-030-29387-1_34 Wallace BC, Small K, Brodley CE et al (2012) Deploying an interactive machine learning system in an evidence-based practice center: Abstrackr. In: Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium. Association for Computing Machinery, New York, NY, USA, IHI ’12, 819–824, https://doi.org/10.1145/2110363.2110464 Wang Y, Gan W, Yang J et al (2019) Dynamic curriculum learning for imbalanced data classification. In: 2019 IEEE/CVF international conference on computer vision (ICCV), 5016–5025, https://doi.org/10.1109/ICCV.2019.00512 Wang X, Pham H, Michel P et al (2020) Optimizing data usage via differentiable rewards. In: III HD, Singh A (eds) Proceedings of the 37th international conference on machine learning, proceedings of machine learning research, vol 119. PMLR, 9983–9995, https://proceedings.mlr.press/v119/wang20p.html Wang X, Chen Y, Zhu W (2021) A survey on curriculum learning. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2021.3069908 Ware M, Frank E, Holmes G et al (2001) Interactive machine learning: letting users build classifiers. Int J Hum Comput Stud 55(3):281–292. https://doi.org/10.1006/ijhc.2001.0499 Weimer M (2010) Machine teaching: a machine learning approach to technology enhanced learning. PhD thesis, Darmstadt University of Technology, http://tuprints.ulb.tu-darmstadt.de/2109/ Weinshall D, Cohen G, Amir D (2018) Curriculum learning by transfer learning: Theory and experiments with deep networks. In: Proceedings of the 35th annual international conference on machine learning, 5235–5243, http://proceedings.mlr.press/v80/weinshall18a.html Weiss K, Khoshgoftaar TM, Wang D (2016) A survey of transfer learning. J Big data 3(1):1–40. https://doi.org/10.1186/s40537-016-0043-6 Weitekamp D, Harpstead E, Koedinger KR (2020) An interaction design for machine teaching to develop AI tutors. In: Proceedings of the 2020 CHI conference on human factors in computing systems, 1–11, https://doi.org/10.1145/3313831.3376226 Wing JM (2021) Trustworthy AI. Commun ACM 64(10):64–71. https://doi.org/10.1145/3448248 Wong JS (2018) Design and fiction: imagining civic AI. Interactions 25(6):42–45. https://doi.org/10.1145/3274568 Xu W (2019) Toward human-centered AI: a perspective from human–computer interaction. Interactions 26(4):42–46. https://doi.org/10.1145/3328485 Xu B, Zhang L, Mao Z et al (2020) Curriculum learning for natural language understanding. In: Proceedings of the 58th annual meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 6095–6104, https://doi.org/10.18653/v1/2020.acl-main.542, https://www.aclweb.org/anthology/2020.acl-main.542 Yang Q, Suh J, Chen NC et al (2018) Grounding interactive machine learning tool design in how non-experts actually build models. In: Proceedings of the 2018 designing interactive systems conference. Association for Computing Machinery, New York, NY, USA, DIS ’18, 573–584, https://doi.org/10.1145/3196709.3196729 Yang SJ, Ogata H, Matsui T et al (2021) Human-centered artificial intelligence in education: seeing the invisible through the visible. Comput Educ 2(100):008. https://doi.org/10.1016/j.caeai.2021.100008 Zbyszynski M, Tanaka A, Visi F (2020) Interactive machine learning: strategies for live performance using electromyography. In: Silva H (ed) Open source biomedical engineering. Springer, Berlin Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. In: Fleet D, Pajdla T, Schiele B et al (eds) European conference on computer vision. Springer, Cham, pp 818–833. https://doi.org/10.1007/978-3-319-10590-1_53 Zeiler MD, Taylor GW, Fergus R (2011) Adaptive deconvolutional networks for mid and high level feature learning. In: 2011 International conference on computer vision, 2018–2025, https://doi.org/10.1109/ICCV.2011.6126474, https://ieeexplore.ieee.org/document/6126474 Zhang X, Kumar G, Khayrallah H et al (2018) An empirical exploration of curriculum learning for neural machine translation. arXiv e-prints arxiv:1811.00739 [cs.CL] Zhang D, Han J, Guo G et al (2019a) Learning object detectors with semi-annotated weak labels. IEEE Trans Circuits Syst Video Technol 29(12):3622–3635. https://doi.org/10.1109/TCSVT.2018.2884173 Zhang X, Shapiro P, Kumar G et al (2019b) Curriculum learning for domain adaptation in neural machine translation. In: Proceedings of the 2019 conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 1903–1915, https://doi.org/10.18653/v1/N19-1189, https://www.aclweb.org/anthology/N19-1189 Zhao Y, Prosperi M, Lyu T et al (2020) Integrating crowdsourcing and active learning for classification of work-life events from tweets. In: Fujita H, Fournier-Viger P, Ali M et al (eds) Trends in artificial intelligence theory and applications. Artificial intelligence practices. Springer, Cham, pp 333–344. https://doi.org/10.1007/978-3-030-55789-8_30 Zhou B, Khosla A, Lapedriza A et al (2016) Learning deep features for discriminative localization. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), 2921–2929, https://doi.org/10.1109/CVPR.2016.319, https://ieeexplore.ieee.org/document/7780688 Zhou Y, Yang B, Wong DF et al (2020) Uncertainty-aware curriculum learning for neural machine translation. In: Proceedings of the 58th annual meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 6934–6944, https://doi.org/10.18653/v1/2020.acl-main.620, https://www.aclweb.org/anthology/2020.acl-main.620 Zhu X (2015) Machine teaching: An inverse problem to machine learning and an approach toward optimal education. In: Proceedings of the twenty-ninth AAAI conference on artificial intelligence. AAAI Press, AAAI’15, 4083–4087, https://ojs.aaai.org/index.php/AAAI/article/view/9761 Zhu X, Singla A, Zilles S et al (2018) An overview of machine teaching. arXiv e-prints arxiv:1801.05927 Zhuang F, Qi Z, Duan K et al (2021) A comprehensive survey on transfer learning. Proc IEEE 109(1):43–76. https://doi.org/10.1109/JPROC.2020.3004555