When Rater Reliability Is Not Enough

Educational Researcher - Tập 41 Số 2 - Trang 56-64 - 2012
Heather C. Hill1, Charalambos Y. Charalambous2, Matthew A. Kraft1
1Harvard Graduate School of Education, Cambridge, MA
2University of Cyprus, Nicosia, Cyprus

Tóm tắt

In recent years, interest has grown in using classroom observation as a means to several ends, including teacher development, teacher evaluation, and impact evaluation of classroom-based interventions. Although education practitioners and researchers have developed numerous observational instruments for these purposes, many developers fail to specify important criteria regarding instrument use. In this article, the authors argue that for classroom observation to succeed in its aims, improved observational systems must be developed. These systems should include not only observational instruments but also scoring designs capable of producing reliable and cost-efficient scores and processes for rater recruitment, training, and certification. To illustrate how such a system might be developed and improved, the authors provide an empirical example that applies generalizability theory to data from a mathematics observational instrument.

Từ khóa


Tài liệu tham khảo

Boston Public Schools. (2010). Evaluation form for teachers. Retrieved from http://www.bostonpublicschools.org

10.1007/978-1-4757-3456-0

10.1080/08957347.2011.532417

Cronbach L. J., 1972, The dependability of behavioral measurements: Theory of generalizability scores and profiles

Danielson Group. (2011). Framework for teaching: Components of professional practice. Retrieved from http://charlottedanielson.com/theframeteach.htm

10.1111/j.1745-3984.1979.tb00081.x

Gordon R., 2006, Identifying effective teachers using performance on the job

10.1023/B:PEEV.0000032427.99952.02

10.1080/07370000802177235

Hill H. C., Herlihy C. (2011). Prioritizing teaching quality in a new system of teacher evaluation. Education Outlook. Retrieved from http://www.aei.org/outlook/101089

10.3102/0002831210387916

Johnson S. M., 1990, Teachers at work: Achieving success in our schools

10.3102/0013189X10390804

10.1080/03054980701782064

10.1007/s10857-010-9140-1

10.1007/BF00151898

Measures of Effective Teaching. (n.d.). Retrieved from the Bill and Melinda Gates Foundation website: http://www.gatesfoundation.org/united-states/Pages/measures-of-effective-teaching-fact-sheet.aspx

National Center for Teacher Effectiveness, 2011, Online poll of states engaged in reform of teacher evaluation systems

10.1016/j.stueduc.2010.10.002

10.3102/01623737026003237

10.3102/00028312024002311

Peterson K. D., 2000, Teacher evaluation: A comprehensive guide to new directions and practices

10.1257/0002828041302244

10.1111/1467-9620.00212

Sartain L., 2009, Evaluation of the excellent in teaching pilot: A report to the Joyce Foundation

Scheerens J., 1997, The foundations of educational effectiveness

Shavelson R. J., 1991, Generalizability theory: A primer

Teacher Evaluation Act, La. Rev. Stat. §§ 17:3881-3886, 17:3901-3905, 17:3997, 17:3891-3895 (2010).

Teddlie C., 2000, The international handbook of school effectiveness research

Tennessee Department of Education. (2011). Framework for evaluation and professional growth. Retrieved from http://www.tn.gov/education/frameval/

Weisberg D., 2009, The widget effect: Our national failure to acknowledge and act on differences in teacher effectiveness

Thông tin
Thông tin xuất bản

Educational Researcher

Tập 41 Số 2

56-64

Thông tin tác giả