Measuring learning that is hard to measure: using the PECSL model to evaluate implicit smart learning

Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. Addison Wesley Longman. Badie, F. (2018). Knowledge building conceptualisation within smart constructivist learning systems. In V. L. Uskov, J. P. Bakken, R. J. Howlett, & L. C. Jain (Eds.), Smart universities: Concepts, systems, and technologies (pp. 385–419). Springer. https://doi.org/10.1007/978-3-319-59454-5_13 Biggs, J. (1995). Assessing for learning: Some dimensions underlying new approaches to educational assessment. The Alberta Journal of Educational Research, 41(1), 1–17. Biggs, J. B., & Collis, K. F. (1982). Evaluating the quality of learning-the SOLO taxonomy (1st ed.). Academic Press. Bloom, B. (1956). Taxonomy of educational objectives, handbook 1: Cognitive domain. Longmans. Bowden, J. (2005). Reflections on the phenomenographic team research project. In J. Bowden & P. Green (Eds.), Doing developmental phenomenography (pp. 11–31). RMIT University Press. Bruner, J. S. (1966). Toward a theory of instruction. Belkapp Press. Carretero, S., Vuorikari, R., & Punie, Y. (2017). DigComp 2.1: The Digital Competence Framework for citizens with eight proficiency levels and examples of use. European Commission. https://doi.org/10.2760/38842 Carroll, J. M., Shih, P. C., Kropczynski, J., Cai, G., Rosson, M. B., & Han, K. (2017). The internet of places at community-scale: Design scenarios for hyperlocal neighborhood. In S. Konomi & G. Roussos (Eds.), Enriching urban spaces with ambient computing, the Internet of Things, and smart city design (pp. 1–24). IGI Global. https://doi.org/10.4018/978-1-5225-0827-4.ch001 Collier-Reed, B. I., Ingerman, A., & Berglund, A. (2009). Reflections on trustworthiness in phenomenographic research: Recognising purpose, context and change in the process of research. Education as Change, 13(2), 339–355. https://doi.org/10.1080/16823200903234901 Cope, C. (2004). Ensuring validity and reliability in phenomenographic research using the analytical framework of a structure of awareness. Qualitative Research Journal, 4(2), 5–18. De Lange, M., & De Waal, M. (2017). Owning the city: New media and citizen engagement in urban design. In K. Etingoff (Ed.), Urban design: Community-based planning (pp. 89–110). Apple Academic Press. Debord, G. (1958). Théorie de la dérive. Internationale Situationniste 2 (Dec. 1958). In Knabb, K. (Ed.), Situationist International Anthology (translation), Bureau of Public Secrets (2006) (pp. 19–23). http://www.bopsecrets.org/SI/2.derive.htm Dron, J. (2018). Smart learning environments, and not so smart learning environments: A systems view. Smart Learning Environments, 5, 25. https://doi.org/10.1186/s40561-018-0075-9 Eraut, M. (2000). Non-formal learning, implicit learning and tacit knowledge. Bristol Policy Press, 4, 12–31. Fang, J. (2013). Colorful robots teach children computer programming: How do you make coding something that kids want to do? Meet Bo and Yana: Covert teaching machines. ZdNET. https://www.zdnet.com/article/colorful-robots-teach-children-computer-programming/ Garrett, J. J. (2010). The elements of user Experience: User-centered design for the web and beyond (2nd ed.). New Riders Press. Gagné, R. M. (1970). The conditions of learning (2nd ed.). Holt. Gee, J. P. (2007). Good video games + good learning. Collected essays on video games, learning and literacy. Peter Lang Publishing. Gibbons, S. (2016). Design thinking 101. Nielsen Norman Group. Green, B. (2019). The smart enough city, putting technology in its place to reclaim our urban future. Strong ideas. MIT Press. Greeno, J. G., & Engeström, Y. (2014). Learning in activity. In R. K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (2nd ed., pp. 128–147). Cambridge University Press. Hense, J., & Mandl, H. (2014). Learning in or with games? In D. Sampson, D. Ifenthaler, J. Spector, & P. Isaias (Eds.), Digital systems for open access to formal and informal learning. Springer. https://doi.org/10.1007/978-3-319-02264-2_12 Hou je Bek, W. (2002). Algorithmic psychogeography. Spacejackers. Hounsell, D. (1984). Essay planning and essay writing. Higher Education Research & Development, 3(1), 13–31. https://doi.org/10.1080/0729436840030102 Hounsell, D. (2005). Contrasting conceptions of essay-writing. In F. Marton, D. Hounsell, & N. Entwistle (Eds.), The experience of learning: Implications for teaching and studying in higher education (3rd ed., pp. 106–125). University of Edinburgh, Institute for Academic Development. Implicit. (2020). Oxford English Dictionary. Retrieved from https://www.lexico.com/definition/implicit Jordan, S. (2015). Writing the smart city: “Relational space” and the concept of “belonging”. Writ. Pract. J. Creative Writ. Res. 1. http://eprints.nottingham.ac.uk/32234/1/WritinginPractice_Version2.pdf Kaapu, T., & Tiainen, T. (2010). User Experience: Consumer Understandings of Virtual Product Prototypes. In K. Kautz & P. A. Nielsen (Eds.), Scandinavian information systems research. First Scandinavian Conference on Information Systems, SCIS 2010, Proceedings. Lecture notes in business information processing (pp. 18–33). Springer. https://doi.org/10.1007/978-3-642-14874-3_2 Kaufman, S. B., DeYoung, C. G., Gray, J. R., Jiménez, L., Brown, J., & Mackintosh, N. (2010). Implicit learning as an ability. Cognition, 116(3), 321–340. https://doi.org/10.1016/j.cognition.2010.05.011 Kazil, P., & Hou je Bek, W. (2010). A walk in the invisible city: World in a shell urban adventure. V2_. https://v2.nl/events/world-in-a-shell-urban-adventure Koehler, A. (2013). Digitizing craft: Creative writing studies and new media: A proposal. College English, 75(4), 379–397. Krathwohl, D. (2002). A revision of Bloom’s taxonomy: An overview. Theory into Practice, 41(4), 212–218. https://doi.org/10.1207/s15430421tip4104_2 Kivunja, C. (2014). Do you want your students to be job-ready with 21st century skills? Change pedagogies: A pedagogical paradigm shift from Vygotskyian social constructivism to critical thinking, problem solving and Siemens’ digital connectivism. International Journal of Higher Education. https://doi.org/10.5430/ijhe.v3n3p81 Lorenzo, N., & Ray Gallon, R. (2019). Smart pedagogy for smart learning. In L. Daniela (Ed.), Didactics of smart pedagogy: Smart pedagogy for technology enhanced learning. Springer. https://doi.org/10.1007/978-3-030-01551-0_3 Lin, T. C. Y. W., Galloway, D., & Lee, W. O. (2011). The effectiveness of action learning in the teaching of citizenship education: A Hong Kong case study. In K. J. Kennedy, W. O. Lee, & D. L. Grossman (Eds.), Citizenship pedagogies in Asia and the Pacific, CERC studies in comparative education (pp. 53–80). Springer. https://doi.org/10.1007/978-94-007-0744-3_4 Lincoln, Y., & Guba, E. (1985). Naturalistic inquiry. Sage Publications. Lister, P. (2019). Learner experience complexity as data variables for smart learning. [Presentation]. AI in Education series, IT Learning Centre, University of Oxford, UK. Lister, P. (2020). Smart learning in the community: Supporting citizen digital skills and literacies. In N. Streitz & S. Konomi (Eds.), Distributed, ambient and pervasive interactions. HCII 2020. Lecture notes in computer science (pp. 533–547). Springer. https://doi.org/10.1007/978-3-030-50344-4_38 Lister, P. (2021a). The pedagogy of experience complexity for smart learning: Considerations for designing urban digital citizen learning activities. Smart Learning Environments. https://doi.org/10.1186/s40561-021-00154-x Lister, P. (2021b). Applying the PECSL: Using case studies to demonstrate the Pedagogy of Experience Complexity for Smart Learning. Smart Learning Environments, 8, 13. https://doi.org/10.1186/s40561-021-00158-7 Lister, P. (2021c). What are we supposed to be learning? Motivation and autonomy in smart learning environments. In N. Streitz & S. Konomi (Eds.), Distributed, ambient and pervasive interactions. HCII 2021. Lecture notes in computer science (Vol. 12782, pp. 235–249). Springer. https://doi.org/10.1007/978-3-030-77015-0_17 Lister, P. (2021d). Understanding experience complexity in a smart learning journey. Springer Nature Social Sciences. https://doi.org/10.1007/s43545-020-00055-9 Lister, P. (2022a). Future-present learning and teaching: a case study in smart learning. In E. Sengupta & P. Blessinger (Eds.), Changing the conventional classroom, Innovations in Higher Education Teaching and Learning (IHETL). Emerald Publishing. Lister, P. (2022b). Ways of experiencing technology in a smart learning environment. In N. Streitz & S. Konomi (Eds.), Distributed, ambient and pervasive interactions. HCII 2022. Lecture notes in computer science. Springer. Liu, D., Huang, R., & Wosinski, M. (2017a). Characteristics and Framework of smart learning. Smart learning in smart cities. Lecture notes in educational technology (pp. 31–48). Springer. Liu D., Huang, R., & Wosinski, M. (2017b). Future trends in smart learning: Chinese perspective. In Smart learning in smart cities. Lecture notes in educational technology (pp. 185–215). Springer. https://doi.org/10.1007/978-981-10-4343-7_8 Marton, F., Dall'Alba, G., & Beaty, E. (1993). Conceptions of learning. In Salo, R (Ed.), Learning discourse: Qualitative research in education (pp. 277–300). International Journal of Educational Research, 19(3), 199–325. Marton, F., & Pong, W. P. (2005). On the unit of description in phenomenography. Higher Education Research & Development, 24(4), 335–348. https://doi.org/10.1080/07294360500284706 Marton, F., Runesson, U., & Tsui, A. B. M. (2004). The space of learning. In F. Marton & A. B. M. Tsui (Eds.), Classroom discourse and the space of learning (pp. 3–40). Lawrence Erlbaum. Marton, F., & Säljö, R. (1976). On qualitative differences in learning: 1. Outcome and process. British Journal of Educational Psychology, 46(1), 4–11. https://doi.org/10.1111/j.2044-8279.1976.tb02980.x Marton, F., & Säljö, R. (2005). Approaches to learning. In F. Marton, D. Hounsell, & N. Entwistle (Eds.), The experience of learning: Implications for teaching and studying in higher education (3 (Internet), pp. 39–58). University of Edinburgh, Centre for Teaching, Learning and Assessment. Marton, F., & Svensson, L. (1979). Conceptions of research in student learning. Higher Education, 8, 471–486. https://doi.org/10.1007/BF01680537 Moreira, F. T., Vairinhos, M., & Ramos, F., et al. (2021). Conceptualization of hypersituation as result of IoT in education. In Ó. Mealha (Ed.), Ludic, co-design and tools supporting smart learning ecosystems and smart education, Proceedings of the 5th international conference on smart learning ecosystems and regional development. Smart innovation, systems and technologies. (Vol. 197). Springer. Newton, G., & Martin, E. (2013). Blooming, SOLO taxonomy, and phenomenography as assessment strategies in undergraduate science education. Journal of College Science Teaching, 43(2), 78–90. Nikolov, R., Shoikova, E., Krumova, M., Kovatcheva, E., Dimitrov, V., & Shikalanov, A. (2016). Learning in a smart city environment. Journal of Communication and Computer, 13, 338–350. https://doi.org/10.17265/1548-7709/2016.07.003 Orgill, M. (2012). Variation theory. In N. M. Seel (Ed.), Encyclopedia of the sciences of learning. Springer. https://doi.org/10.1007/978-1-4419-1428-6_272 O’Riordan, T., Millard, D. E., & Schulz, J. (2016). How should we measure online learning activity? Research in Learning Technology. https://doi.org/10.3402/rlt.v24.30088 Paterson, R. (2019). The power of EMPs: Educational multimedia projects. In L. Daniela (Ed.), Didactics of smart pedagogy: Smart pedagogy for technology enhanced learning (pp. 393–414). Springer. https://doi.org/10.1007/978-3-030-01551-0_20 Pask, G., & Scott, B. C. E. (1972). Learning strategies and individual competence. International Journal of Man-Machine Studies, 4, 217–253. Pérez-Mateo, M., Maina, M., Guitert, M., & Romero, M. (2011). Learner generated content: Quality criteria in online collaborative learning. The European Journal of Open, Distance and E-Learning, 14. https://www.eurodl.org/?p=special&sp=articles&article=459 Pinder, D. (2005). Arts of urban exploration. Cultural Geographies, 12(4), 383–411. Prensky, M. (2003). Digital game-based learning. ACM Computers in Entertainment, 1(1), 21. Reber, A. S. (1989). Implicit learning and tacit knowledge. Journal of Experimental Psychology, 118(3), 219–235. Reed, B. (2006). Phenomenography as a way to research the understanding by students of technical concepts. Núcleo de Pesquisa em Tecnologia da Arquitetura e Urbanismo (NUTAU): Technological Innovation and Sustainability, São Paulo, Brazil (pp. 1–11). Remnant, F., & Avard, R. (2016). Qualitative Impact Assessment Protocol (QUIP). BetterEvaluation. Retrieved from http://betterevaluation.org/en/plan/approach/QUIP Rezgui, K., Mhiri, H., & Ghédira, K. (2014). An ontology-based profile for learner representation in learning networks. International Journal of Emerging Technologies in Learning (iJET). https://doi.org/10.3991/ijet.v9i3.3305 Richardson, J. (1999). The Concepts and Methods of Phenomenographic Research. Review of Educational Research, 69(1), 53–82. https://doi.org/10.3102/00346543069001053. Roisko, H. (2007). Adult learners’ learning in a university setting a phenomenographic study. [Doctoral Dissertation, University of Tampere]. Tampere University Press. http://urn.fi/urn:isbn:978-951-44-6928-2 Sacré, H., de Droogh, L., De Wilde, A., & De Visscher, S. (2017). Storytelling in urban spaces: Exploring storytelling as a social work intervention in processes of urbanisation. In H. Sacré & S. De Visscher (Eds.), Learning the city, cultural approaches to civic learning in urban spaces (pp. 35–49). Springer. https://doi.org/10.1007/978-3-319-46230-1_3 Sacré, H., & De Visscher, S. (2017). A cultural perspective on the city. In H. Sacré & S. De Visscher (Eds.), Learning the city, cultural approaches to civic learning in urban spaces (pp. 1–17). Springer. https://doi.org/10.1007/978-3-319-46230-1_1 Saffer, D. (2010). Designing for interaction, second edition: Creating innovative applications and devices. New Riders. Sandberg, J. (2005). How Do We Justify Knowledge Produced Within Interpretive Approaches? Organizational Research Methods, 8(1), 41–68. https://doi.org/10.1177/1094428104272000. Säljö, R. (1979a). Learning in the learner’s perspective: Some commonplace misconceptions. Reports from the Institute of Education, University of Gothenburg. Säljö, R. (1979b). Learning about learning. Higher Education, 8, 443–451. https://doi.org/10.1007/BF01680533 Schmeck, R. R. (Ed.). (1988). Learning strategies and learning styles. Perspectives on individual differences. Springer. https://doi.org/10.1007/978-1-4899-2118-5 Seger, C. A. (1994). Implicit learning. Psychological Bulletin, 115(2), 163–196. Selwyn, N. (2011). Education and technology: Key issues and debates. Continuum. Siemens, G. (2006). Knowing knowledge. Internet Archive. Available from https://archive.org/details/KnowingKnowledge/ Sjöström, B., & Dahlgren, L. O. (2002). Applying phenomenography in nursing research. Journal of Advanced Nursing, 40(3), 339–345. https://doi.org/10.1046/j.1365-2648.2002.02375.x Slater, M. (2017). Implicit learning through embodiment in immersive virtual reality. In D. Liu, C. Dede, R. Huang, & J. Richards (Eds.), Virtual, augmented, and mixed realities in education. Smart computing and intelligence (pp. 19–33). Springer. Souleles, N., Savva, S., Watters, H., Annesley, A., & Bull, B. (2014). A phenomenographic investigation on the use of iPads among undergraduate art and design students. British Journal of Educational Technology, 46(1), 131–141. https://doi.org/10.1111/bjet.1213 Svensson, L. (1997). Theoretical foundations of phenomenography. Higher Education Research and Development, 16(2), 159–172. Taylor, C., & Cope, C. J. (2007). Are there educationally critical aspects in the concept of evolution? [Paper presentation]. UniServe Science: Science Teaching and Learning Symposium, University of Sydney, held September 28–29, 2007. https://openjournals.library.sydney.edu.au/index.php/IISME/article/view/6352/6991 Uskov, V. L., Bakken, J. P., Pandey, A., Singh, U., Yalamanchili, M., & Penumatsa, A. (2016). Smart university taxonomy: Features, components, systems. In V. Uskov, R. Howlett, & L. Jain (Eds.), Smart education and e-learning 2016. Smart innovation, systems and technologies. (Vol. 59). Springer. https://doi.org/10.1007/978-3-319-39690-3_1 Vinod Kumar, T. M. (2020). Smart environment for smart cities. In T. M. Vinod Kumar (Ed.), Smart environment for smart cities, advances in 21st century human settlements (pp. 1–53). Springer. https://doi.org/10.1007/978-981-13-6822-6_1 Webb, G. (1997). Deconstructing deep and surface: Towards a critique of phenomenography. Higher Education, 33, 195–212. Wegerif, R. (2022). Beyond democracy: Education as design for dialogue. In Liberal democratic education: A paradigm in crisis (pp. 157–179). Brill mentis (pre-print). Wilson-Grau, R. (2015). Outcome harvesting. BetterEvaluation. Retrieved from http://betterevaluation.org/plan/approach/outcome_harvesting