Gillard and Garrett Ignore the Evidence Against Teacher BonusesTuesday May 31, 2011
Nothing more clearly demonstrates that education policy under the Gillard Government is an evidence-free zone than the announcement last week on new funding to start paying bonuses to the top teachers in Australia.
The Government has proceeded with the scheme even though the large majority of major research studies in recent years, including three new academic studies in recent months, demonstrate that teacher bonuses do not increase student achievement and can actually undermine student learning.
The Prime Minister, the Treasurer and the Minister for Education jointly announced that bonuses will be paid to about one in ten teachers – around 25,000 teachers. The first bonuses will be based on performance in the 2013 school year and paid in early 2014. Teachers with the most experience will get about $8,100 and teachers in the first few years of their career will get around $5,400 each. The total cost will be $1.3 billion over eight years.
Bonus payments will be based on lesson observations, student performance data (including NAPLAN and school based information that can show the valued added by particular teachers), parental feedback and teacher qualifications and professional development undertaken.
Several recent studies have demonstrated that the bonuses are unlikely to achieve their goal of improving student results.
A review of studies of three major performance pay schemes operating in the United States found that they have had no effect on student test scores in reading, mathematics and science. The review is published in the latest issue of the National Tax Journal.
The studies reviewed used randomized designs and rigorous quasi-experimental designs which involved comparing student results for schools and teachers included in the programs with matched groups not included in the programs. Each scheme involved the payment of bonuses to teachers based on performance in student results. In each case, there was no overall difference in results between students whose teachers were included in the programs and those whose teachers were not included.
In contrast, the review notes that several studies in countries such as India and Kenya report generally positive effects of performance pay on student achievement. However, it is less clear whether these programs actually promoted long-run learning, as some studies find the effects do not persist over time or document opportunistic behaviours on the part of treatment teachers that account for increased student achievement. Furthermore, the incentive structure facing teachers and schools in the countries are very different from the operational context found within the US and Australian public school systems.
A study published last March by Harvard University economist, Roland Fryer, evaluated the New York teacher bonus scheme introduced by former Schools Chancellor Joel Klein and found that it failed to increase student achievement. Fryer said in the abstract to the paper:
I find no evidence that teacher incentives increase student performance, attendance, or graduation, nor do I find any evidence that the incentives change student or teacher behaviour. If anything, teacher incentives may decrease student achievement, especially in larger schools.
The study concludes:
Providing incentives to teachers based on school’s performance on metrics involving student achievement, improvement, and the learning environment did not increase student achievement in any statistically meaningful way. If anything, student achievement declined. [p.5]
A recent study by Derek Neal, Professor of Economics at the University of Chicago, published by the National Bureau of Economic Research concludes that tying teacher performance pay to student test scores in unlikely to improve education. Neal says that recent teacher performance pay schemes have failed because they often rely on measures that were never intended to help determine teacher pay.
Many accountability and performance pay systems employ test scores from assessment systems that produce information used not only to determine rewards and punishments for educators but also to inform the public about secular progress in student learning. As long as education authorities keep trying to accomplish both of these tasks with one set of assessments, they will continue to fail at both tasks. [p.3]
Neal concludes that:
....accountability systems always create predictable effort distortions when employed as incentive systems. Systems that serve as mechanisms for providing public information about the achievement of students and the performance of schools relative to public education standards distort effort if they contain rewards or sanctions that provide incentives for educators. [pp. 9-10]
Neal notes that several studies provide persuasive evidence that the measured gains induced by a particular performance pay program represented little or no improvement in actual subject mastery as teachers often resort to coaching students. Coaching involves activities such as practising tests that improve scores on a given assessment without improving student mastery of a subject. Teachers may avoid teaching that leads to more comprehensive learning while increasing time devoted to activities that prepare students for upcoming assessments.
For example, they may reduce writing assignments in favour of having students find mistakes in prepared passages. In mathematics it may involve working only on problems that follow a format or rubric which is known to be tested. Such changes in teaching practice harm the development of writing and mathematical skills among students, but it makes sense as a strategy for raising test scores on standardized tests.
Neal also notes research evidence that some teachers resort to cheating by helping students with answers during tests or changing answers. When coaching or cheating happens, the tests cease to work as measures of student achievement or teacher competence.
Another problem is that problem is that rewarding teachers by student test scores encourages the best teachers to seek transfers to schools with high-performing students, abandoning the students who need them most. Research shows that teacher incentive programs based on student achievement have encouraged teachers to leave schools in which most students come from disadvantaged backgrounds.
Neal concludes with the following recommendation:
Thus, it seems obvious that a key task for those who design future performance pay schemes for teachers is the creation of a series of assessments that consistently cover a well specified curriculum but vary substantially in terms of specific item content and format. Put more pointedly, the designers of assessment-based incentive schemes must take seriously the challenge of designing a series of assessments such that the best response of educators is not to coach but to teach in ways that build true mastery of the intended domain. [p.45]
These findings are supported by a range of earlier studies. But, despite this large weight of evidence, advocates of bonus payments now rely on a cross-country study published last year by the Harvard Program on Education Policy and Governance that shows positive effects on student achievement. The study found that nations that pay teachers on their performance score higher on the OECD’s PISA international tests after controlling for a range of background variables that influence student results.
However, a recent review of this study questions its findings and conclusions. It makes several criticisms of the study. For example, there is considerable variability across countries in how performance pay is determined as definitions and applications of performance pay vary widely from country to country. However, the study fails to properly consider fundamental differences among countries in the types of performance pay system. Nations are simply lumped together as having or not having a performance pay plan. The review says that this variability in definitions makes the interpretation of the results of the study somewhat problematic. Although the same term (performance pay) may be applied to these diverse definitions and practices, it is unlikely that the same thing is being measured across countries.
The review says that the small sample size of 28 countries is a problem because small differences in have a large effect on the correlations. For example, the inclusion or exclusion of a single country results in large shifts in the size of the reported relationships. That is, the numbers become unreliable and invalid.
The review also criticises the lack of sophistication in the way background variables are treated in the study. It says the study should have taken account of the likely possibility of variations in relationships of background variables and student outcomes between countries. The study used inadequate statistical techniques to take account of such variation.
The review notes that, besides performance pay, there are a number of other relevant variables with a significant relationship to the average performance of a country that are not considered in the analysis. It may be, for example, that high-scoring countries experiment with performance pay rather than high scores being caused by merit pay. Consequently, “causal interpretations based on its analysis should be avoided” [p.6]. It also noted other factors that caution against generalizing from the findings of the report.
The review concludes that “drawing conclusions about performance pay from this analysis cannot be validly sustained” [p. 7] and is not warranted. It says:
More analyses and more in-depth studies of differences among educational systems are needed before a decision to adopt performance pay should even be discussed. [p.7]
So, the teacher bonus scheme promoted by Julia Gillard and Peter Garrett should never have got off the ground. There is no evidence to support it. It is yet another example of the faith-based approach to education policy by the Gillard Government.
For the review of evaluations of teacher performance pay schemes in the United States and elsewhere see:
Michael Podgursky & Mathew Springer, Teacher Compensation Systems in the United States K-12 Public School System, National Tax Journal, March 2011, 64 (1): 165–192.
For the evaluation of the New York scheme see:
Roland G. Fryer, Teacher Incentives and Student Achievement: Evidence from New York City Public Schools, Working Paper No. 16850, National Bureau of Economic Research, Cambridge, Mass, March 2011.
For the NBER study see:
Derek Neal, The Design of Performance Pay in Education, Working Paper No. 16710, National Bureau of Economic Research, Cambridge, Mass, March 2011.
For the Harvard study see:
Ludger Woessmann, Cross-Country Evidence on Teacher Performance Pay, PEPG 10-11, Harvard Program on Education Policy and Governance, Cambridge, MA, June 2010.
An abridged version of the paper is published in the journal Education Next:
Ludger Woessmann, Merit Pay International, Education Next, Spring 2011, 11 (2): 73-77.
For the review of the Harvard study see:
Mathias von Davier, Review of Cross-Country Evidence on Teacher Performance Pay, National Education Policy Center, School of Education, University of Colorado, Boulder, March 2011.
For an overview of other research studies on teacher performance pay see the SOS Research Brief: Teacher bonuses and student achievement.pdf.
Commenting is closed for this article.