“I can see that”: Developing shared rubric category interpretations through score negotiation

Anton, 2009, Dynamic assessment of advanced second language learners, Foreign Language Annals, 42, 576, 10.1111/j.1944-9720.2009.01030.x Bahktin, 1981 Beckner, 2009, Language is a complex adaptive system: Position paper, Language Learning, 59, 1 Bond, 2007 Bonk, 2003, A many-facet Rasch analysis of the second language group oral discussion task, Language Testing, 20, 89, 10.1191/0265532203lt245oa Brown, 2011, Variables that affect the dependability of L2 pragmatics tests, Journal of Pragmatics, 43, 198, 10.1016/j.pragma.2010.07.026 Creswell, 2011 East, 2009, Evaluating the reliability of a detailed analytic scoring rubric for foreign language writing, Assessing Writing, 14, 88, 10.1016/j.asw.2009.04.001 Eckes, 2005, Examining rater effects in TestDaF writing and speaking performance assessments: A many-facet Rasch analysis, Language Assessment Quarterly: An International Journal, 2, 197, 10.1207/s15434311laq0203_2 Eckes, 2011 Goldberg, 2012, Constructionist approaches, 15 Hodges, 1992, Values as constraints on affordances: Perceiving and acting properly, Journal for the Theory of Social Behaviour, 22, 263, 10.1111/j.1468-5914.1992.tb00220.x Hodges, 2007, Good prospects: Ecological and social perspectives on conforming, creating, and caring in conversation, Language Sciences, 29, 584, 10.1016/j.langsci.2007.01.003 Hodges, 2007, Values define fields: The intentional dynamics of driving, carrying, leading, negotiating, and conversing, Ecological Psychology, 19, 153 Hodges, 2009, Ecological pragmatics: Values, dialogical arrays, complexity, and caring, Pragmatics & Cognition, 17, 628, 10.1075/pc.17.3.08hod Jacobs, 1981 Janssen, 2015, Building a better rubric: Mixed methods rubric revision, Assessing Writing, 26, 51, 10.1016/j.asw.2015.07.002 Johnson, 2009, The influence of rater language background on writing performance assessment, Language Testing, 26, 485, 10.1177/0265532209340186 Johnson, 2000, The relation between score resolution methods and interrater reliability: An empirical study of an analytic scoring rubric, Applied Measurement in Education, 13, 121, 10.1207/S15324818AME1302_1 Johnson, 2001, Score resolution and the interrater reliability of holistic scores in rating essays, Written Communication, 18, 229, 10.1177/0741088301018002003 Johnson, 2005, Resolving score differences in the rating of writing samples: Does discussion improve the accuracy of scores?, Language Assessment Quarterly: An International Journal, 2, 117, 10.1207/s15434311laq0202_2 Kane, 2006, Validation, 17 Kane, 2012, Validating score interpretations and uses, Language Testing, 29, 3, 10.1177/0265532211417210 Kane, 2013, Validating the interpretations and uses of test scores, Journal of Educational Measurement, 50, 1, 10.1111/jedm.12000 Knoch, 2011, Investigating the effectiveness of individualized feedback to rating behavior: A longitudinal study, Language Testing, 28, 179, 10.1177/0265532210384252 Kondo-Brown, 2002, A FACETS analysis of rater bias in measuring Japanese second language writing performance, Language Testing, 19, 3, 10.1191/0265532202lt218oa Meier, 2016, Principled rubric adoption and adaptation: One mulit-method case study, 165 Lantolf, 2010, Dynamic assessment in the classroom: Vygotskian praxis for second language development, Language Testing, 15, 11 Lantolf, 2000 Lantolf, 2006 Lim, 2011, The development and maintenance of rating quality in performance writing assessment: A longitudinal study of new and experienced raters, Language Testing, 28, 543, 10.1177/0265532211406422 Linacre, 2010 Linacre, J. (n.d.). Estimate considerations. Retrieved on 05.10.15 from http://www.winsteps.com/facetman/estimationconsiderations.html. Lumley, 1995, Rater characteristics and rater bias: Implications for training, Language Testing, 12, 54, 10.1177/026553229501200104 Lynch, 1998, Using G-theory and Many-facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants, Language Testing, 15, 158, 10.1177/026553229801500202 McKelvey, 1988, Differences in conformity among friends and strangers, Psychological Reports, 62, 759, 10.2466/pr0.1988.62.3.759 McNamara, 1996 Peirce, 1998, Vol. 2 Poehner, 2005, Dynamic assessment in the language classroom, Language Teaching Research, 9, 233, 10.1191/1362168805lr166oa Poehner, 2007, Beyond the test: L2 dynamic assessment and the transcendence of mediated learning, Modern Language Journal, 9, 323, 10.1111/j.1540-4781.2007.00583.x Rutherford, 1987 Saldaña, 2013, An introduction to codes and coding, 1 Schaefer, 2008, Rater bias patterns in an EFL writing assessment, Language Testing, 25, 465, 10.1177/0265532208094273 Swales, 2012 Trace, 2015, Measuring the impact of rater negotiation in writing performance assessment, Language Testing Upshur, 1995, Constructing rating scales for second language tests, ELT Journal, 49, 3, 10.1093/elt/49.1.3 Van Lier, 2004 Weigle, 1998, Using FACETS to model rater training effects, Language Testing, 15, 263, 10.1177/026553229801500205 Weigle, 1999, Investigating rater/prompt interactions in writing assessment: Quantitative and qualitative approaches, Assessing Writing, 6, 145, 10.1016/S1075-2935(00)00010-6 1995, Sociocultural studies of mind Wigglesworth, 1993, Exploring bias analysis as a tool for improving rater consistency in assessing oral interaction, Language Testing, 10, 305, 10.1177/026553229301000306 Wray, 2012, What do we (Think we) know about formulaic language? An evaluation of the current state of play, Annual Review of Applied Linguistics, 32, 231, 10.1017/S026719051200013X