Criteria and Bias in Native English Teachers’ Assessment of L2 Pragmatic Appropriacy: Content and FACETS Analyses

citation_journal_title=RELC Journal; citation_title=Understanding discrepancies in rater judgement on national-level oral examination tasks; citation_author=HT Ang-Aw, CCM Goh; citation_volume=42; citation_issue=1; citation_publication_date=2011; citation_pages=31-51; citation_doi=10.1177/0033688210390226; citation_id=CR1 citation_journal_title=Language Testing; citation_title=‘Naïve’ native speakers and judgments of oral proficiency in Spanish; citation_author=D Barnwell; citation_volume=6; citation_issue=2; citation_publication_date=1989; citation_pages=152-163; citation_doi=10.1177/026553228900600203; citation_id=CR2 citation_journal_title=Language Testing; citation_title=The effect of rater variables in the development of an occupation-specific language performance test; citation_author=A Brown; citation_volume=12; citation_issue=1; citation_publication_date=1995; citation_pages=1-15; citation_doi=10.1177/026553229501200101; citation_id=CR3 citation_title=Understanding language structure, interaction, and variation; citation_publication_date=2000; citation_id=CR4; citation_author=S Brown; citation_author=S Attardo; citation_publisher=The University of Michigan Press citation_title=Politeness; citation_publication_date=1987; citation_id=CR5; citation_author=P Brown; citation_author=S Levinson; citation_publisher=Cambridge University Press citation_journal_title=Intercultural Pragmatics; citation_title=From ‘sorry very much’ to ‘I’m ever so sorry’: Acquisitional patterns in L2 apologies by Catalan learners of English; citation_author=SM Dalmau, CH Gotor; citation_volume=4; citation_issue=2; citation_publication_date=2007; citation_pages=287-315; citation_doi=10.1515/IP.2007.014; citation_id=CR6 Du, Y., Wright, B. D., & Brown, W. L. (1996). Differential facet functioning detection in direct writing assessment. Paper presented at the Annual Meeting of the American Educational Research Association, New York. citation_journal_title=Language Assessment Quarterly; citation_title=Examining rater effects in TestDaF writing and speaking performance assessments: A many-facet Rasch analysis; citation_author=T Eckes; citation_volume=2; citation_issue=3; citation_publication_date=2005; citation_pages=197-221; citation_doi=10.1207/s15434311laq0203_2; citation_id=CR8 citation_journal_title=Language Testing; citation_title=Rater types in writing performance assessments: A classification approach to rater variability; citation_author=T Eckes; citation_volume=25; citation_issue=2; citation_publication_date=2008; citation_pages=155-185; citation_doi=10.1177/0265532207086780; citation_id=CR9 citation_journal_title=Language Testing; citation_title=Evaluating rater response to an online training program for L2 writing assessment; citation_author=C Elder, G Barkhuizen, U Knock, J Randow; citation_volume=24; citation_issue=1; citation_publication_date=2007; citation_pages=37-64; citation_doi=10.1177/0265532207071511; citation_id=CR10 citation_journal_title=Journal of Educational Measurement; citation_title=Examining rater errors in the assessment of written compositions with a many-faceted Rasch model; citation_author=G Englehard; citation_volume=31; citation_issue=2; citation_publication_date=1994; citation_pages=93-112; citation_doi=10.1111/j.1745-3984.1994.tb00436.x; citation_id=CR11 citation_title=On apologizing; citation_inbook_title=Conversational routine: Explorations in standardized communication situations and prepatterned speech; citation_publication_date=1981; citation_pages=259-271; citation_id=CR12; citation_author=B Fraser; citation_publisher=Mouton citation_journal_title=Modern Language Journal; citation_title=Perceptions of the communicative efforts of American students of Spanish; citation_author=VB Galloway; citation_volume=64; citation_issue=4; citation_publication_date=1980; citation_pages=428-433; citation_doi=10.1111/j.1540-4781.1980.tb05218.x; citation_id=CR13 citation_journal_title=Language in Society; citation_title=Apologies in New Zealand English; citation_author=J Holmes; citation_volume=19; citation_issue=2; citation_publication_date=1990; citation_pages=155-199; citation_doi=10.1017/S0047404500014366; citation_id=CR14 citation_journal_title=System; citation_title=Teachers’ perceptions of error: The effects of first language and experience; citation_author=K Hyland, E Anan; citation_volume=34; citation_issue=4; citation_publication_date=2006; citation_pages=509-519; citation_doi=10.1016/j.system.2006.09.001; citation_id=CR15 citation_journal_title=Language Testing; citation_title=The influence of rater language background on writing performance assessment; citation_author=JS Johnson, GS Lim; citation_volume=26; citation_issue=4; citation_publication_date=2009; citation_pages=485-505; citation_doi=10.1177/0265532209340186; citation_id=CR16 citation_journal_title=Journal of Pragmatics; citation_title=The semantic and pragmatic analysis of South Korean and Australian English apologetic speech acts; citation_author=H Kim; citation_volume=40; citation_issue=2; citation_publication_date=2008; citation_pages=257-278; citation_doi=10.1016/j.pragma.2007.11.003; citation_id=CR17 citation_journal_title=Proceedings of Sophia University Linguistic Society; citation_title=Longitudinal study on the development of pragmatic competence in a natural learning context: Perception behind performance; citation_author=S Kondo; citation_volume=12; citation_publication_date=1997; citation_pages=35-54; citation_id=CR18 Kondo, S. (1997b). The development of pragmatic competence by Japanese learners of English in a natural learning context: Longitudinal study on interlanguage apologies, MA Thesis, Sophia University, Graduate School of Language and Linguistics. citation_journal_title=Language Testing; citation_title=A FACETS analysis of rater bias in measuring Japanese L2 writing performance; citation_author=K Kondo-Brown; citation_volume=19; citation_issue=1; citation_publication_date=2002; citation_pages=3-31; citation_doi=10.1191/0265532202lt218oa; citation_id=CR20 citation_journal_title=Asia Pacific Education Review; citation_title=Native and non-native rater behavior in grading Korean students’ English essays; citation_author=HK Lee; citation_volume=10; citation_issue=3; citation_publication_date=2009; citation_pages=387-397; citation_doi=10.1007/s12564-009-9030-3; citation_id=CR21 citation_journal_title=Language Testing; citation_title=The development and maintenance of rating quality in performance writing assessment: A longitudinal study of new and experienced raters; citation_author=GS Lim; citation_volume=28; citation_issue=4; citation_publication_date=2011; citation_pages=543-560; citation_doi=10.1177/0265532211406422; citation_id=CR22 citation_title=FACETS; citation_publication_date=1996; citation_id=CR23; citation_author=JM Linacre; citation_author=B Wright; citation_publisher=MESA Press citation_journal_title=Reflections on English Language Teaching; citation_title=Assessing EFL learners’ interlanguage pragmatic knowledge: Implications for testers and teachers; citation_author=J Liu; citation_volume=5; citation_issue=1; citation_publication_date=2006; citation_pages=1-22; citation_id=CR24 citation_journal_title=Language Testing; citation_title=Assessment criteria in a large-scale writing test: What do they really mean to the raters?; citation_author=T Lumley; citation_volume=19; citation_issue=3; citation_publication_date=2002; citation_pages=246-276; citation_doi=10.1191/0265532202lt230oa; citation_id=CR25 citation_title=Assessing second language writing: The rater’s perspective; citation_publication_date=2005; citation_id=CR26; citation_author=T Lumley; citation_publisher=Peter Lang citation_journal_title=Language Testing; citation_title=Rater characteristics and rater bias: Implications for training; citation_author=T Lumley, TF McNamara; citation_volume=12; citation_issue=1; citation_publication_date=1995; citation_pages=54-71; citation_doi=10.1177/026553229501200104; citation_id=CR27 citation_title=Transfer and proficiency in interlanguage apologizing; citation_inbook_title=Speech acts across cultures; citation_publication_date=1996; citation_pages=155-187; citation_id=CR28; citation_author=N Maeshiba; citation_author=N Yoshinaga; citation_author=G Kasper; citation_author=S Ross; citation_publisher=Mouton de Gruyter McIntyre, P. N. (1993). The importance and effectiveness of moderation training on the reliability of teachers’ assessment of ESL writing samples. MA Thesis, University of Melbourne. citation_title=Measuring second language performance; citation_publication_date=1996; citation_id=CR30; citation_author=TF McNamara; citation_publisher=Longman citation_journal_title=Journal of Hokkaido University of Education at Kushiro; citation_title=Apology is not necessary: An in-depth analysis of my own intercultural and intracultural miscommunication; citation_author=K Nonaka; citation_volume=32; citation_publication_date=2000; citation_pages=155-186; citation_id=CR31 citation_title=Apology: A speech act set; citation_inbook_title=Sociolinguistics and language acquisition; citation_publication_date=1983; citation_pages=18-35; citation_id=CR32; citation_author=E Olshtain; citation_author=AD Cohen; citation_publisher=Newbury House citation_journal_title=System; citation_title=The FCE Speaking test: using rater reports to help interpret test scores; citation_author=M Orr; citation_volume=30; citation_issue=2; citation_publication_date=2002; citation_pages=143-154; citation_doi=10.1016/S0346-251X(02)00002-7; citation_id=CR33 citation_journal_title=Language Testing; citation_title=A multi-method analysis of evaluation criteria used to assess the speaking proficiency of graduate student instructors; citation_author=IC Plough, SL Briggs, S Bonn; citation_volume=27; citation_issue=2; citation_publication_date=2010; citation_pages=235-260; citation_doi=10.1177/0265532209349469; citation_id=CR34 citation_journal_title=Organizational Behavior and Human Decision Processes; citation_title=Why convene rater teams: an investigation of the benefits of anticipated discussion, consensus, and rater motivation; citation_author=SG Roch; citation_volume=104; citation_issue=1; citation_publication_date=2007; citation_pages=14-29; citation_doi=10.1016/j.obhdp.2006.08.003; citation_id=CR35 Roever, C. (2001). A web-based test of interlanguage pragmalinguistic knowledge: Speech acts, routines, and implicatures. Unpublished doctoral Dissertation, University of Hawai‘i, Honolulu, Hawai‘i. citation_journal_title=Language Testing; citation_title=Rater bias patterns in an EFL writing assessment; citation_author=E Schaefer; citation_volume=25; citation_issue=4; citation_publication_date=2008; citation_pages=465-493; citation_doi=10.1177/0265532208094273; citation_id=CR37 citation_journal_title=Modern Language Journal; citation_title=The effect of raters’ background and training on the reliability of direct writing tests; citation_author=E Shohamy, CM Gordon, R Kraemer; citation_volume=76; citation_issue=1; citation_publication_date=1992; citation_pages=27-33; citation_doi=10.1111/j.1540-4781.1992.tb02574.x; citation_id=CR38 Son, M. (2010). Examining rater bias: An evaluation of possible factors influencing elicited imitation ratings. MA project, Brigham Young University. citation_journal_title=Journal of Second Language Writing; citation_title=Do English and ESL faculty differ in evaluating the essays of native English-speaking and ESL students?; citation_author=B Song, I Caruso; citation_volume=5; citation_issue=2; citation_publication_date=1996; citation_pages=163-182; citation_doi=10.1016/S1060-3743(96)90023-5; citation_id=CR40 citation_journal_title=Communication Research; citation_title=A Japan–U.S. comparison of apology styles; citation_author=N Sugimoto; citation_volume=24; citation_issue=4; citation_publication_date=1997; citation_pages=349-370; citation_doi=10.1177/009365097024004002; citation_id=CR41 citation_title=Longitudinal studies in interlanguage pragmatics; citation_inbook_title=Handbook of pragmatics; citation_publication_date=2010; citation_pages=333-361; citation_id=CR42; citation_author=N Taguchi; citation_publisher=Mouton de Gruyter citation_journal_title=Pragmatics; citation_title=Rater variation in the assessment of speech acts; citation_author=N Taguchi; citation_volume=21; citation_issue=3; citation_publication_date=2011; citation_pages=453-471; citation_id=CR43 Weigle, S.C. (1994a). Effects of training on raters of English as a second language compositions: Quantitative and qualitative approaches. Ph.D. Dissertation, University of California, Los Angeles. citation_journal_title=Language Testing; citation_title=Effects of training on raters of ESL compositions; citation_author=SC Weigle; citation_volume=11; citation_issue=2; citation_publication_date=1994; citation_pages=197-223; citation_doi=10.1177/026553229401100206; citation_id=CR45 citation_journal_title=Language Testing; citation_title=Using FACETS to model rater training effects; citation_author=SC Weigle; citation_volume=15; citation_issue=2; citation_publication_date=1998; citation_pages=263-287; citation_doi=10.1177/026553229801500205; citation_id=CR46 citation_journal_title=Language Testing; citation_title=Exploring bias analysis as a tool for improving rater consistency in assessing oral interaction; citation_author=G Wigglesworth; citation_volume=10; citation_issue=3; citation_publication_date=1993; citation_pages=305-335; citation_doi=10.1177/026553229301000306; citation_id=CR47 citation_journal_title=Australian Review of Applied Linguistics; citation_title=Patterns of rater behaviour in the assessment of an oral interaction test; citation_author=G Wigglesworth; citation_volume=17; citation_issue=2; citation_publication_date=1994; citation_pages=77-103; citation_id=CR48 citation_title=Investigating interlanguage pragmatic ability; citation_inbook_title=Investigating pragmatics in foreign language learning, teaching and testing; citation_publication_date=2008; citation_pages=201-223; citation_id=CR49; citation_author=S Yamashita; citation_publisher=Multilingual Matters citation_journal_title=Second Language Studies; citation_title=Rater bias in assessing the pragmatics of KFL learners using facets analysis; citation_author=SJ Youn; citation_volume=26; citation_issue=1; citation_publication_date=2007; citation_pages=85-163; citation_id=CR50 citation_journal_title=Language Testing; citation_title=Judgments of oral proficiency by non-native and native English speaking teacher raters: Competing or complementary constructs?; citation_author=Y Zhang, C Elder; citation_volume=28; citation_issue=1; citation_publication_date=2011; citation_pages=31-50; citation_doi=10.1177/0265532209360671; citation_id=CR51