Multimodal data capabilities for learning: What can multimodal data tell us about learning?

British Journal of Educational Technology - Tập 51 Số 5 - Trang 1450-1484 - 2020
Kshitij Sharma, Michail N. Giannakos

Tóm tắt

AbstractMost research on learning technology uses clickstreams and questionnaires as their primary source of quantitative data. This study presents the outcomes of a systematic literature review of empirical evidence on the capabilities of multimodal data (MMD) for human learning. This paper provides an overview of what and how MMD have been used to inform learning and in what contexts. A search resulted in 42 papers that were included in the analysis. The results of the review depict the capabilities of MMD for learning and the ongoing advances and implications that emerge from the employment of MMD to capture and improve learning. In particular, we identified the six main objectives (ie, behavioral trajectories, learning outcome, learning‐task performance, teacher support, engagement and student feedback) that the MMLA research has been focusing on. We also summarize the implications derived from the reviewed articles and frame them within six thematic areas. Finally, this review stresses that future research should consider developing a framework that would enable MMD capacities to be aligned with the research and learning design (LD). These MMD capacities could also be utilized on furthering theory and practice. Our findings set a baseline to support the adoption and democratization of MMD within future learning technology research and development. Practitioner NotesWhat is already known about this topic Capturing and measuring learners’ engagement and behavior using MMD has been explored in recent years and exhibits great potential. There are documented challenges and opportunities associated with capturing, processing, analyzing and interpreting MMD to support human learning. MMD can provide insights into predicting learning engagement and performance as well as into supporting the process. What this paper adds Provides a systematic literature review (SLR) of empirical evidence on MMD for human learning. Summarizes the insights MMD can give us about the learning outcomes and process. Identifies challenges and opportunities of MMD to support human learning. Implications for practice and/or policy Learning analytics researchers will be able to use the SLR as a guide for future research. Learning analytics practitioners will be able to use the SLR as a summary of the current state of the field.

Từ khóa


Tài liệu tham khảo

Ahn T. B., 2020, Exploring emotions and multimodal learning analytics: Eye‐tracking and facial recognition, British Journal of Educational Technology

10.1007/s11257-019-09233-8

10.1145/3027385.3027429

10.18608/jla.2017.43.3

10.18608/jla.2016.32.14

10.1016/j.ijhcs.2009.12.003

Baker R. S., 2015, The Oxford handbook of affective computing, 233

10.1145/2883851.2883944

10.1111/jcal.12268

10.1111/bjet.12983

10.1177/1088868318772990

10.1145/2460296.2460316

10.1007/s10758-016-9291-y

10.18608/jla.2016.32.11

10.1109/T-AFFC.2010.1

10.18608/jla.2017.42.14

10.1007/s11257-009-9062-8

10.1111/bjet.12959

10.1111/bjet.13015

10.1111/bjet.12829

10.1016/j.compedu.2017.08.007

D’Mello S. K., 2005, Workshop at the International Conference on Intelligent User Interfaces, 7

10.1080/00461520.2017.1281747

10.1007/s11257-010-9074-4

10.1016/j.learninstruc.2011.10.001

10.1145/2682899

10.1007/978-3-642-13388-6_29

10.1007/978-3-030-23204-7_9

10.1145/3027385.3027447

10.1145/3303772.3303776

10.1111/jcal.12288

10.1111/bjet.12981

10.1016/S0304-3940(98)00805-2

10.1111/jcal.12291

10.1016/j.infsof.2008.01.006

10.1111/bjet.12992

10.1145/2723576.2723589

10.1145/2723576.2723588

10.1109/TLT.2013.18

Georgiadis K., 2015, International Conference on Games and Learning Alliance, 517

10.1016/j.ijinfomgt.2019.02.003

Giannakos M., 2020, Companion Proceedings of LAK20

10.1109/FG.2018.00037

10.1007/s11257-017-9188-z

10.1007/978-3-319-19773-9_70

10.1145/3025453.3025808

10.1109/TLT.2017.2754497

10.1007/s11257-019-09228-5

10.1046/j.1460-9568.2002.01975.x

10.1111/jcal.12262

10.1016/j.compedu.2014.11.016

Kidzinski Ł., 2016, Proceedings of the Ninth International Conference on Educational Data Mining, 406

Kitchenham B., 2007, Guidelines for performing systematic literature reviews in software engineering

10.1145/3170427.3170620

10.1007/s11528-018-0294-5

10.1111/bjet.12958

10.1145/3242587.3242642

10.18608/jla.2018.51.4

10.1111/jcal.12315

10.1109/TLT.2018.2868673

10.18608/jla.2018.53.7

10.1145/3303772.3303825

10.1007/s11042-009-0344-2

10.1145/3170358.3170379

10.1145/3027385.3027401

10.1145/2883851.2883873

10.18608/jla.2017.41.4

10.1145/2993148.2993202

Muñoz‐Cristóbal J. A., 2017, Joint Proceedings of the Sixth Multimodal Learning Analytics (MMLA) Workshop and the Second Cross‐LAK Workshop Co‐located with Seventh International Learning Analytics and Knowledge Conference, 60

Newell A., 1994, Unified Theories Of Cognition

10.18608/jla.2018.53.8

Nicol D., 2013, Feedback in higher and professional education: understanding it and doing it well, 44

10.1109/ACCESS.2018.2876801

10.1016/j.chb.2018.12.019

10.1111/bjet.12987

10.1145/3170358.3170406

10.1111/bjet.12982

10.1109/HICSS.2016.14

Paquette L., 2016, Sensor‐free or sensor‐full: A comparison of data modalities in multi‐channel affect detection

10.1109/TLT.2016.2639508

10.1109/IVS.2017.7995948

10.1016/j.chb.2018.11.008

10.4018/978-1-59140-562-7.ch034

10.1145/2883851.2883927

10.1111/jcal.12232

10.1109/TSMC.1983.6313160

10.1016/j.compedu.2018.01.015

10.1145/3170358.3170364

10.1007/978-3-319-66610-5_13

10.1016/j.ergon.2005.04.005

10.1109/TASL.2010.2101596

10.1109/ICALT.2014.234

10.1016/j.compedu.2011.05.013

10.1207/s1532690xci1604_4

10.1109/ICALT.2018.00057

Sharma K., 2020, Assessing cognitive performance using physiological and facial features: Generalizing across contexts, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 4, 10.1145/3411811

10.1111/bjet.12854

Sharma K., 2019, Thirteenth International Conference on Computer Supported Collaborative Learning (CSCL), 684

10.1145/2883851.2883920

10.1111/jcal.12263

10.1145/3242969.3242989

10.1145/3341162.3348383

Vujovic M., 2020, Studying collaborative learning and space design with multimodal learning analytics, British Journal of Educational Technology

Worsley M., 2018, Multimodal learning analytics’ past, present, and potential futures

10.1145/2723576.2723624

10.1007/s40593-017-0160-1

10.1007/s11257-019-09241-8

Thông tin
Thông tin xuất bản

British Journal of Educational Technology

Tập 51 Số 5

1450-1484

Thông tin tác giả