Quantitative content analysis procedures to analyse students’ reflective essays: A methodological review of psychometric and edumetric aspects

∗Abou Baker El-Dib, 2007, Levels of reflection in action research. An overview and an assessment tool, Teaching and Teacher Education, 23, 24, 10.1016/j.tate.2006.04.002 Baartman, L. K. J. (2008). ‘Assessing the assessment’ Development and use of quality criteria for Competence Assessment Programmes (Doctoral dissertation). Retrieved from: http://igitur-archive.library.uu.nl/dissertations/2008-0423-200627/UUindex.html. Baartman, 2006, The wheel of competency assessment: Presenting quality criteria for competency assessment programs, Studies In Educational Evaluation, 32, 153, 10.1016/j.stueduc.2006.04.006 Baartman, 2007, Evaluating assessment quality in competence-based education: A qualitative comparison of two frameworks, Educational Research Review, 2, 114, 10.1016/j.edurev.2007.06.001 ∗Bain, 1999, Using journal writing to enhance student teachers’ reflectivity during field experience placements, Teachers and Teaching, 5, 51, 10.1080/1354060990050104 ∗Ballard, K. K. (2006). Using Van Manen’s model to assess levels of reflectivity among preservice physical education teachers (Doctoral dissertation). Retrieved from: http://txspace.tamu.edu/bitstream/1969.1/4373/1/etd-tamu-2006B-KINE-Ballard.pdf. Biggs, 2003, Aligning teaching and assessing to course objectives, Teaching and Learning in Higher Education: New Trends and Innovations, 2, 13 Birenbaum, 2007, Evaluating the assessment: Sources of evidence for quality assurance, Studies In Educational Evaluation, 33, 29, 10.1016/j.stueduc.2007.01.004 Borsboom, 2004, The concept of validity, Psychological Review, 111, 1061, 10.1037/0033-295X.111.4.1061 Boud, 1990, Assessment and the promotion of academic values, Studies in Higher Education, 15, 101, 10.1080/03075079012331377621 Boud, D. (2006). Relocating reflection in the context of practice: Rehabilitation or rejection? Paper presented at the SCUTREA 2006: 36th Annual Conference, Leeds, UK. Boud, 1998, Promoting reflection in professional courses: The challenge of context, Studies in Higher Education, 23, 191, 10.1080/03075079812331380384 Broekman, 1999, Teacher development by using writing as a tool, Teacher development, 3, 233, 10.1080/13664539900200079 Carver, 1974, Two dimensions of tests: Psychometric and edumetric, American Psychologist, 29, 512, 10.1037/h0036782 ∗Chamoso, 2009, Analysis of the reflections of student-teachers of mathematics when working with learning portfolios in Spanish university classrooms, Teaching and Teacher Education, 25, 198, 10.1016/j.tate.2008.09.007 Clarke, 1996, Reflective practice: Reviewing the issues and refocusing the debate, International Journal of Nursing Studies, 33, 171, 10.1016/0020-7489(95)00045-3 Crooks, 1996, Threats to the valid use of assessments, Assessment in Education: Principles, Policy & Practice, 3, 265, 10.1080/0969594960030302 ∗Davis, 2006, Characterizing productive reflection among preservice elementary teachers: Seeing what matters, Teaching and Teacher Education, 22, 281, 10.1016/j.tate.2005.11.005 De Wever, 2006, Content analysis schemes to analyze transcripts of online asynchronous groups: A review, Computers & Education, 46, 6, 10.1016/j.compedu.2005.04.005 Dierick, 2001, New lines in edumetrics: New forms of assessment lead to new assessment criteria, Studies In Educational Evaluation, 27, 307, 10.1016/S0191-491X(01)00032-3 Dierick, 2002, De actuele kwaliteit van asssessment: Ontwikkelingen in de edumetrie [the actual quality of assessment: Developments in the edumetry] Dochy, 2001, A new assessment era: Different needs, new challenges, Research Dialogue in Learning and Instruction, 2, 11, 10.1016/S0959-4752(00)00022-0 Dochy, 1997, Assessment as a tool for learning, Studies In Educational Evaluation, 23, 279, 10.1016/S0191-491X(97)86211-6 Embretson, 2007, Construct validity: A universal validity system or just another test evaluation procedure?, Educational Researcher, 36, 449, 10.3102/0013189X07311600 Glaser, 1990, Toward new models for assessment, International Journal of Educational Research, 14, 475 Gulikers, 2008, Authenticity is in the eye of the beholder: Student and teacher perceptions of assessment authenticity, Journal of Vocational Education & Training, 60, 401, 10.1080/13636820802591830 ∗Hamann, H. M. (2002). Reflective practices and confluent educational perspectives: Three exploratory studies. Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans. Harrison, 2005, Facilitating the professional learning of new teachers through critical reflection on practice during mentoring meetings, European Journal of Teacher Education, 28, 267, 10.1080/02619760500269392 Hayes, 2007, Answering the call for a standard reliability measure for coding data, Communication Methods and Measures, 1, 77, 10.1080/19312450709336664 Hennissen, 2008, Mapping mentor teachers’ roles in mentoring dialogues, Educational Research Review, 3, 168, 10.1016/j.edurev.2008.01.001 Hoover, 1994, Reflective writing as a window on preservice teachers’ thought processes, Teaching and Teacher Education, 10, 83, 10.1016/0742-051X(94)90042-6 Kane, 2008, Terminology, emphasis, and utility in validation, Educational Researcher, 37, 76, 10.3102/0013189X08315390 ∗Kember, 1999, Determining the level of reflective thinking from students’ written journals using a coding scheme based on the work of Mezirow, International Journal of Lifelong Education, 18, 18, 10.1080/026013799293928 ∗Kember, 2008, A four-category scheme for coding and assessing the level of reflection in written work, Assessment & Evaluation in Higher Education, 33, 369, 10.1080/02602930701293355 Kolb, 1984 Korthagen, 1999, Linking theory and practice. Changing the pedagogy of teacher education, Educational Researcher, 28, 4, 10.3102/0013189X028004004 Krippendorff, 2004, Reliability in content analysis, Human Communication Research, 30, 411 Lee, 2005, Understanding and assessing preservice teachers’ reflective thinking, Teaching and Teacher Education, 21, 699, 10.1016/j.tate.2005.05.007 Linn, 1991, Complex, performance-based assessment: Expectations and validation criteria, Educational Researcher, 20, 15, 10.3102/0013189X020008015 Lombard, 2002, Content analysis in mass communication: Assessment and reporting of intercoder reliability, Human Communication Research, 28, 587, 10.1111/j.1468-2958.2002.tb00826.x ∗Maclellan, 2004, How reflective is the academic essay?, Studies in Higher Education, 29, 281, 10.1080/1234567032000164886 ∗Mansvelder-Longayroux, 2007, The portfolio as a tool for stimulating reflection by student teachers, Teaching and Teacher Education, 23, 47, 10.1016/j.tate.2006.04.033 ∗McAlpine, 1999, Building a metacognitive model of reflection, Higher Education, V37, 105, 10.1023/A:1003548425626 ∗McAlpine, 2004, Reflection on teaching: Types and goals of reflection, Educational Research and Evaluation, 10, 337, 10.1080/13803610512331383489 ∗McCollum, S. (1997). Insights into the process of guiding reflection during an early field experience of preservice teachers (Doctoral dissertation). Retrieved from: http://scholar.lib.vt.edu/theses/available/etd-302283039731191/. Messick, 1989, Meaning and values in test validation: The science and ethics of assessment, Educational Researcher, 18, 5, 10.3102/0013189X018002005 Messick, 1995, Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning, American Psychologist, 50, 741, 10.1037/0003-066X.50.9.741 Mezirow, 1990, How critical reflection triggers transformative learning, 1 Mezirow, 1991 Mezirow, 1998, On critical reflection, Adult Education Quarterly, 48, 185, 10.1177/074171369804800305 Mislevy, 2007, Validity by design, Educational Researcher, 36, 463, 10.3102/0013189X07311660 Moss, 2007, Reconstructing validity, Educational Researcher, 36, 470, 10.3102/0013189X07311608 Nummedal, 1996, Assessment and reflective teaching, Innovative Higher Education, V21, 39, 10.1007/BF01238010 ∗Pee, 2002, Appraising and assessing reflection in students’ writing on a structured worksheet, Medical Education, 36, 575, 10.1046/j.1365-2923.2002.01227.x Peltier, 2005, The reflective learning continuum: Reflecting on reflection, Journal of Marketing Education, 27, 250, 10.1177/0273475305279657 ∗Plack, 2005, A method for assessing reflective journal writing, Journal of Allied Health, 34, 199 Rourke, 2004, Validity in quantitative content analysis, Educational Technology Research and Development, 52, 5, 10.1007/BF02504769 Schön, D. A. (1987). Educating the reflective practitioner. Paper presented at the American Educational Research Association, Washington, DC. Shepard, 1993, Evaluating test validity, Review of Research in Education, 19, 405 Shin, 2006, Learning to teach writing through tutoring and journal writing, Teachers and Teaching: Theory and Practice, 12, 325, 10.1080/13450600500467621 Spalding, 2002, Demistifying reflection: A study of pedagogical strategies that encourage reflective journal writing, Teachers College Record, 104, 1393, 10.1111/1467-9620.00208 ∗Sparks-Langer, 1990, Reflective pedagogical thinking: How can we promote it and measure it?, Journal of Teacher Education, 41, 23, 10.1177/002248719004100504 Stokking, 2004, Teachers’ assessment of students’ research skills, British Educational Research Journal, 30, 93, 10.1080/01411920310001629983 Strijbos, 2006, Content analysis: What are they talking about?, Computers & Education, 46, 29, 10.1016/j.compedu.2005.04.002 Strijbos, 2007, Methodological issues in developing a multi-dimensional coding procedure for small-group chat communication, Learning and Instruction, 17, 394, 10.1016/j.learninstruc.2007.03.005 Thorpe, 2004, Reflective learning journals: From concept to practice, Reflective Practice, 5, 327, 10.1080/1462394042000270655 ∗Tsangaridou, 1994, Using pedagogical reflective strategies to enhance reflection among preservice physical education teachers, Teaching in Physical Education, 14, 13, 10.1123/jtpe.14.1.13 Valcke, 2006, The problem arena of researching computer supported collaborative learning: Introduction to the special section, Computers & Education, 46, 1, 10.1016/j.compedu.2005.04.004 Van der Schaaf, M. F. (2005). Construct validation of teacher portfolio assessment. Procedures for improving teacher competences assessment illustrated by teaching students research skills (Doctoral dissertation). Utrecht University, Utrecht. Van der Schaaf, 2005, Cognitive representations in raters’ assessment of teacher portfolios, Studies In Educational Evaluation, 31, 27, 10.1016/j.stueduc.2005.02.005 Van der Schaaf, 2008, Developing and validating a design for teacher portfolio assessment, Assessment & Evaluation in Higher Education, 33, 245, 10.1080/02602930701292522 Wallman, 2008, A categorization scheme for assessing pharmacy students” levels of reflection during internships, American Journal of Pharmaceutical Education, 72, 1, 10.5688/aj720105 ∗Wong, 1995, Assessing the level of student reflection from reflective journals, Journal of Advanced Nursing, 22, 48, 10.1046/j.1365-2648.1995.22010048.x York-Barr, 2006, Reflective practice for continuous learning Yost, 2000, An examination of the construct of critical reflection: Implications for teacher education programming in the 21st century, Journal of Teacher Education, 51, 39, 10.1177/002248710005100105