Identifying design values across countries through adaptive comparative judgment

Alamir, A. (2015). Comparative Judgment Update. Retrieved February 23, 2018, from http://digitalassess.com/comparative-judgement-update/. Bartholomew, S. R. (2017). Assessing open-ended design problems. Technology and Engineering Education Teacher,76(6), 13–17. Bartholomew, S., Hartell, E., & Strimel, G. (2017a). ACJ: A tool for international assessment collaboration. In PATT34 Millersville University, Pennsylvania, USA 10–14 July, 2017. Bartholomew, S. R., Reeve, E., Veon, R., Goodridge, W., Stewardson, G., Lee, V., et al. (2017b). Mobile devices, self-directed learning, and achievement in Technology and Engineering Education classrooms during a STEM activity. Journal of Technology Education,29(1), 2–24. Bartholomew, S. R., Strimel, G. S., & Jackson, A. (2017c). A comparison of traditional and adaptive comparative judgment assessment techniques for freshmen engineering design projects. International Journal of Engineering,34(1), 20–33. (Manuscript submitted for publication). Bartholomew, S. R., Strimel, G. S., & Yoshikawa, E. (2018). Using adaptive comparative judgment for student formative feedback and learning during a middle school open-ended design challenge. International Journal of Technology & Design Education. https://doi.org/10.1007/s10798-018-9442-7. Bartholomew, S. R., Zhang, L., Garcia Bravo, E., & Strimel, G. J. (2019). A tool for formative assessment and learning in a graphics design course: Adaptive comparative judgment. The Design Journal. https://doi.org/10.1080/14606925.2018.1560876. Bouncken, R., Brem, A., & Kraus, S. (2016). Multi-cultural teams as sources for creativity and innovation: The role of cultural diversity on team performance. International Journal of Innovation Management,20(01), 1650012. Bramley, T. (2015). Investigating the reliability of adaptive comparative judgment (Vol. 36). Cambridge: Cambridge Assessment. Charmaz, K., & Belgrave, L. (2012). Qualitative interviewing and grounded theory analysis. The SAGE Handbook of Interview Research: The Complexity of the Craft, 2, 347–365. Cunningham, C. M. (2009). Engineering is elementary. The Bridge,30(3), 11–17. de la Paz, K. B. (2017). Who are we? (special issue). Technology and Engineering Teacher, 76(4), 8. De Vries, M. J. (2015). Research challenges for the future. In P. Williams, A. Jones, & C. Buntting (Eds.), The future of technology education: Contemporary issues in technology education (pp. 253–269). Singapore: Springer. Denson, C. D., Buelin, J. K., Lammi, M. D., & D’Amico, S. (2015). Developing instrumentation for assessing creativity in engineering design. Journal of Technology Education,27(1), 23–40. Department of Education. (2013). National curriculum in England: Design and technology programmes of study. Retrieved from https://www.gov.uk/government/publications/nationalcurriculum-in-england-design-and-technology-programmes-of-study. Accessed 25 Feb 2019. Design and Technology Association. (2017). Designed and made in Britain…? Retrieved from https://www.data.org.uk/campaign/. Accessed 25 Feb 2019. Diefes-Dux, H. A., Moore, T., Zawojewski, J., Imbrie, P. K., & Follman, D. (2004). A framework for posing open-ended engineering problems: Model-eliciting activities. In 34th Annual frontiers in education, 2004. FIE 2004 (pp. F1A–3). IEEE. DigitalAssess. (2018). What we do: CompareAssess. Retrieved from http://digitalassess.com/what-wedo/#compareassess. Discover McDonald’s Around the Globe: McDonald’s (2010). Retrieved August 25, 2017, from http://corporate.mcdonalds.com/mcd/country/map.html. Accessed 25 Feb 2019. Dutson, A. J., Todd, R. H., Magleby, S. P., & Sorensen, C. D. (1997). A review of literature on teaching engineering design through project-oriented capstone courses. Journal of Engineering Education,86(1), 17–28. Edens, K. M. (2000). Preparing problem solvers for the 21st century through problem-based learning. College Teaching,48(2), 55–60. Friedman, T. L. (2005). The world is flat: A brief history of the twenty-first century. London: Macmillan. Gay, G. (2002). Preparing for culturally responsive teaching. Journal of teacher education,53(2), 106–116. Getto, G., & Sun, H. (2017). Localizing user experience: Strategies, practices, and techniques for culturally sensitive design. Technical Communication,64(2), 89–94. Gilpin, R. (2016). The political economy of international relations. Princeton: Princeton University Press. Guidry Lacina, J. (2002). Preparing international students for a successful social experience in higher education. New Directions for Higher Education,2002(117), 21–28. Hagberg, J. E., & Hultén, M. (2005). Skolans undervisning och elevers lärande i teknik: svensk forskning i internationell kontext. Stockholm: Vetenskapsrådet. Hallström, J. (2009). Technical knowledge in a technical society: Elementary school technology education in Sweden, 1919–1928. History of Education,38(4), 455–474. https://doi.org/10.1080/00467600801995427. Hartell, E. (2015). Assidere necesse est: Necessities and complexities regarding teachers’ assessment practices in technology education. Doctoral thesis. KTH Royal Institute of Technology. Retrieved from http://www.diva-portal.org/smash/record.jsf?pid=diva2%3A788413&dswid=-3318. Accessed 25 Feb 2019. Hartell, E., & Skogh, I. B. (2015). Criteria for success: A study of primary technology teachers’ assessment of digital portfolios. Australasian Journal of Technology Education,2(1), 1–17. Hultén, M. (2013). Technology for all: Turning a keyword into a school subject in post-war Sweden. History of Education,42(5), 622–637. https://doi.org/10.1080/0046760X.2013.832410. Idler, S. (2013). How to design for a cross-cultural user experience. Retrieved August 25, 2017, from https://usabilla.com/blog/designing-for-a-cross-cultural-user-experience-part1/. Accessed 25 Feb 2019. International Technology and Engineering Educators Association (ITEA/ITEEA). (2000/2002/2007). Standards for technological literacy: Content for the study of technology. Reston, VA: Author. Jones, I., & Adcock, L. (2012). Summative peer assessment of undergraduate calculus using adaptive comparative judgement. In P. Iannone & A. Simpson (Eds.), Mapping university mathematics assessment practices. Norwich: University of East Anglia. Jones, I., & Wheadon, C. (2015). Peer assessment using comparative and absolute judgement. Studies in Educational Evaluation, 47, 93–101. Katehi, L., Pearson, G., & Feder, M. (2009). Engineering in K-12 education, Committee on K-12 Engineering Education, National Academy of Engineering and National Research Council of the National Academies. Khan, T., Pitts, M., & Williams, M. A. (2016). Cross-cultural differences in automotive HMI design: A comparative study between UK and Indian users’ design preferences. Journal of Usability Studies,11(2), 45–65. Kimbell, R. (2007). E-assessment in project e-scape. Design & Technology Education: An International Journal,12(2), 66–76. Kimbell, R. (2012a). Evolving project e-scape for national assessment. International Journal of Technology and Design Education,22, 135–155. Kimbell, R. (2012b). The origins and underpinning principles of e-scape. International Journal of Technology and Design Education,22, 123–134. Kimbell, R., & Stables, K. (2007). Researching design learning: Issues and findings from two decades of research and development (Vol. 34). Berlin: Springer. Kimbell, R., Wheeler, T., Stables, K., Shepard, T., Martin, F., Davies, D., et al. (2009). e-scape portfolio assessment: A research & development project for the Department of Children, Families and Schools, phase 3 report (p. 169). London: Goldsmiths, University of London. Lee, J. J., & Sayed, S. (2008). Culturally sensitive design. In Y. Levanto, P. Sivenius, & S. Vihma (Eds.), Design connections: Knowledge, value and involvement through design (pp. 54–63). Finland: UIAH Press. Lewis, T., & Zuga, K. (2005). A conceptual framework of ideas and issues in technology education. Washington, DC: National Science Foundation. Lindsay, B., & Simeon, E. J. (2016). University social and public engagement: Creative nexuses for STEM research and international relations. In Globalisation and higher education reforms (pp. 59–75). Springer. Montgomery, C., & McDowell, L. (2009). Social networks and the international student experience: An international community of practice? Journal of Studies in International Education,13(4), 455–466. Moye, J. J., Jones, V. R., & Dugger, W. E. (2015). Status of technology and engineering education in the United States: A fifth report of the findings from the states (2014-15). Technology and Engineering Teacher,74(7), 30–36. NAE. (2011/2016). The Swedish National Agency for Education Curriculum for the compulsory school, preschool class and the recreation center, 2011 (2011). Sweden. Retrieved from http://www.skolverket.se/publikationer?id=2687. Accessed 25 Feb 2019. National Academy of Engineering (NAE) & National Research Council (NRC). (2014). STEM integration in K–12 education: Status, prospects, and an agenda for research. Washington, DC: National Academies Press. Newhouse, C. P. (2014). Using digital representations of practical production work for summative assessment. Assessment in Education: Principles, Policy & Practice,21(2), 205–220. Nikander, P. (2008). Working with transcripts and translated data. Qualitative Research in Psychology,5(3), 225–231. Norström, P. (2014). Technological knowledge and technology education. Doctoral thesis. KTH Royal Institute of Technology. Retrieved from http://kth.diva-portal.org/smash/record.jsf?pid=diva2%3A715009&dswid=62. ISBN: 978-91-7595-078-5. Accessed 25 Feb 2019. National Research Council (NRC). (2009). Engineering in K–12 education: Understanding the status and improving the prospects. Washington, DC: National Academies Press. National Research Council (NRC). (2012). Education for life and work: Developing transferable knowledge and skills in the 21st Century. Washington: National Academies Press. Partnership for 21st Century Skills. (2017). P21 framework definitions. Washington, DC: Author. Retrieved from http://www.p21.org/our-work/p21-framework. Pollitt, A. (2004). Let’s stop marking exams. Retrieved from http://www.cambridgeassessment.org.uk/images/109719-let-s-stop-marking-exams.pdf. Accessed 25 Feb 2019. Pollitt, A. (2012). The method of adaptive comparative judgment. Assessment in Education: Principles, Policy & Practice,19(3), 281–300. Pollitt, A. (2015). On ‘Reliability’ bias in ACJ. Cambridge Exam Research. Retrieved 2 Feb 2018. https://www.researchgate.net/publication/283318012_On_'Reliability'_bias_in_ACJ. Pollitt, A., & Crisp, V. (2004). Could comparative judgments of script quality replace traditional marking and improve the validity of exam questions? Paper presented at the BERA annual conference, UMIST Manchester, England. Qualifications and Curriculum Authority. (2000). Design and technology in the national curriculum. London: HMSO. Retrieved from http://www.qca.org.uk. Accessed 25 Feb 2019. Rangel, C., & Lynch, D. (2018). Addressing the issue of bias in the measurement of reliability in the method of Adaptive Comparative Judgment. Athlone, Ireland: Paper presented at the 36th Pupils’ Attitudes towards Technology Conference. Rasch, G. (1960/1980). Probabilistic models for some intelligence and attainment tests. Copenhagen: Danish Institute for Educational Research. (Reprinted as 2nd ed., 1980, Chicago: University of Chicago Press). Rasinen, A. (2003). An analysis of the technology education curriculum of six countries. Journal of Technology Education, 15. Retrieved from http://scholar.lib.vt.edu/ejournals/JTE/v15n1/rasinen.html. Accessed 25 Feb 2019. Regeringskansliet. (2017). Stärkt digital kompetens i skolans styrdokument. Promemoria 20170309. Stockholm, Sweden. Retrieved from https://www.regeringen.se/pressmeddelanden/2017/03/starkt-digital-kompetens-i-laroplaner-ochkursplaner/. Accessed 25 Feb 2019. Rowsome, P., Seery, N., & Lane, D. (2013). The development of pre-service design educator’s capacity to make professional judgments on design capability using adaptive comparative judgment. American Society for Engineering Education. Saldaña, J. (2009). The coding manual for qualitative researchers. Thousand Oaks, CA: SAGE. Schilling, K., & Applegate, R. (2012). Best methods for evaluating educational impact: A comparison of the efficacy of commonly used measures of library instruction. Journal of the Medical Library Association,100(4), 258–269. https://doi.org/10.3163/1536-5050.100.4.007. Seery, N., & Canty, D. (2017). Assessment and learning: The proximal and distal effects of comparative judgment. In Handbook of technology education (pp. 1–14). Seery, N., Canty, D., & Phelan, P. (2012). The validity and value of peer assessment using adaptive comparative judgement in design driven practical education. International Journal of Technology and Design Education,22(2), 205–226. Seery, N., Delahunty, T., Canty, D., & Buckley, J. (2017). Illustrating educational development through ipsative performance in design based education. In PATT2017: Technology & engineering education–fostering the creativity of youth around the globe. Seery, N., Delahunty, T., Sorby, S., & Sadowski, M., (2018). Adaptive Comparative Judgment as an alternative to the Delphi method for establishing a concept inventory for graphics. Paper presented at the ASEE EDGD Annual Conference, Kingston, Jamaica. Shah, N. (2013). Cross cultural consideration for user interface design. Retrieved August 25, 2017, from http://www.humanfactors.com/newsletters/cross-cultural-considerations-for-user-interface-design.asp. Accessed 25 Feb 2019. Skolinspektionen. (2014). (Swedish School Inspectorate). Teknik – gör det osynliga synligt Om kvaliteten i grundskolans teknikundervisning. Stockholm. Skolverket/The Swedish National Agency for Education. (2018). Curriculum for the compulsory school, preschool class, and the recreation centre. Retrieved from https://www.skolverket.se/sitevision/proxy/publikationer/svid12_5dfee44715d35a5cdfa2899/55935574/wtpub/ws/skolbok/wpubext/trycksak/Blob/pdf3984.pdf?k=3984. Accessed 25 Feb 2019. Starkweather, K. N. (2015). Politics and policy. In P. Williams, A. Jones, & C. Buntting (Eds.), The future of technology education: Contemporary issues in technology education (pp. 239–252). Singapore: Springer. Steedle, J. T., & Ferrara, S. (2016). Evaluating comparative judgment as an approach to essay scoring. Applied Measurement in Education,29(3), 211–223. Strimel, G. (2014). Authentic education by providing a situation for student-selected problem-based learning. Technology and Engineering Teacher, 73(7), 8–18. Strimel, G., & Grubbs, M. E. (2016). Positioning technology and engineering education as a key force in STEM education. Journal of Technology Education,27(2), 21–36. Strimel, G. J., Grubbs, M. E., & Wells, J. G. (2016). Engineering education: A clear decision. Technology and Engineering Teacher,76(4), 18–24. Tarricone, P., & Newhouse, C. P. (2016). Using comparative judgement and online technologies in the assessment and measurement of creative performance and capability. International Journal of Educational Technology in Higher Education,13(1), 16. https://doi.org/10.1186/s41239-016-0018-x. Teknikföretagen, & Cetis, (2012). Teknikämnet i träda: teknikföretagens och CETIS rapport om teknikundervisningen i grundskolan. Stockholm: Teknikföretagen. Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review,34, 273–286. van Daal, T., Lesterhuis, M., Coertjens, L., Donche, V., & De Maeyer, S. (2016). Validity of comparative judgement to assess academic writing: examining implications of its holistic character and building on a shared consensus. Assessment in Education: Principles, Policy & Practice, 26(1), 59–74. https://doi.org/10.1080/0969594X.2016.1253542. Whitehouse, C., & Pollitt, A. (2012). Using adaptive comparative judgement to obtain a highly reliable rank order in summative assessment. Retrieved on February 23, 2018 from https://cerp.aqa.org.uk/sites/default/files/pdf_upload/CERP_RP_CW_20062012_2.pdf. Accessed 25 Feb 2019. Williams, P. J., & Kimbell, R. (2012). E-scape (special issue). International Journal of Technology and Design Education, 22(2), 265–270.