Teacher Compensation Experiment in India
Given the wildly different social conditions, it’s hardly clear how relevant it is to U.S. policy disputes, but Karthik Muralidharan and Venkatesh Sundararaman report:
Performance pay for teachers is frequently suggested as a way of improving education outcomes in schools, but the theoretical predictions regarding its effectiveness are ambiguous and the empirical evidence to date is limited and mixed. We present results from a randomized evaluation of a teacher incentive program implemented across a large representative sample of government-run rural primary schools in the Indian state of Andhra Pradesh. The program provided bonus payments to teachers based on the average improvement of their students’ test scores in independently administered learning assessments (with a mean bonus of 30% of monthly pay). At the end of two years of the program, students in incentive schools performed significantly better than those in control schools by 0.28 and 0.16 standard deviations in math and language tests respectively. They scored significantly higher on “conceptual” as well as “mechanical” components of the tests, suggesting that the gains in test scores represented an actual increase in learning outcomes. Incentive schools also performed better on subjects for which there were no incentives, suggesting positive spillovers. Group and individual incentive schools performed equally well in the first year of the program, but the individual incentive schools outperformed in the second year. Incentive schools performed significantly better than other randomly-chosen schools that received additional schooling inputs of a similar value.
The spillover point is especially interesting. There are a number of subjects for which we don’t have very good testing mechanisms, and obviously you don’t want to base compensation on a bad test. But pretty good tests for basic reading and math are available, and in an optimistic scenario enhanced achievement in those two core skill areas will help kids do better in other subjects as well. That seems to have been the case in this experiment. At the federal level, the Obama administration’s proposed increase in Teacher Incentive Fund funding is the relevant policy fight in this area.
36 Responses to “Teacher Compensation Experiment in India”
Register
RSSLogin
Topic Cloud
ArchivesBlog RollAbout Matt YglesiasContact Me
Donate
September 10th, 2009 at 9:30 am
My favorite part of this post? The city labeled on the map is essentially named “Hydra Bad.” Yes, India, many-headed serpents are indeed bad.
September 10th, 2009 at 9:38 am
haven’t test-based incentives been tried here in the u.s.? isn’t that what the whole “teaching the test” problem is all about?
September 10th, 2009 at 9:41 am
Before the standard education fireworks start, I’d just point out the MY started a few posts ago by saying that merit paying wasn’t about encouraging teachers to teach better, it was about using financial incentives to keep good teachers in the system. This post seems entirely about using incentives to get teachers to teach better.
To the point, I think I’m like a lot of people on this subject: open to the idea but extremely wary about implementation. Any attempts to reward teachers based on test results are going to have to have serious barriers in place to avoid the games Texas played with test results: especially expelling kids to boost average scores and playing with test structure in order to boost scores rather than measure achievement (ever wonder why schools starts so early in standardized testing states: it’s because adding a week of test prep without moving the testing dates is the easiest way to get a boost in scores).
September 10th, 2009 at 9:45 am
It’s amazing that Matt can perseverate on this point after what we just saw about Obama’s speech to the school children. It seems obvious that our problems with education run a little deeper than a few underperforming teachers.
Another thing Matt doesn’t seem to grasp is that our public schools are supposed to present equality of opportunity for the students. As obviously imperfect as this is today, with some good teachers and some bad teachers, I don’t see how it gets anything but worse if some children go to classes with teachers who get paid more and others go to classes with teachers who get paid less. And I don’t need a crystal ball to see the potential for real corruption here.
Gee, I wonder how well those students would do if they knew the top third of the test-takers could attend college for free? I’m guessing that might make more difference in test scores than some hypothetical Mr. Chips.
Matt is illustrating the cowardice and corruption of conservo-liberal philosophy. He never talks about the real changes that need to be made, like proper nutrition and health care, stable families, and small classes, because those things are expensive and would upset the oligarchy. He always talks about more money and freedom for administration and teachers (but no union members need apply!) because the first rule is that any funding must never go to the actual client.
It’s a philosophy that has served us poorly in other social arenas and is now chipping away at the public schools.
September 10th, 2009 at 9:45 am
So Arne Duncan is going to set my wife up with a $20,000 bonus? Cool.
September 10th, 2009 at 9:48 am
Improvement-based performance tests are a canard. Kids from low-income environments not only start out below high-income kids, they don’t improve nearly as fast. Kids spend only a small fraction of their time in school and there is only so much teachers can do. How stupid do you have to be to think that low-income students in public schools do worse than high-income students in public schools because they have bad teachers?
Pupils of teachers in rich districts and private schools will improve faster and those teachers will get the money. It is true that throwing money at schools in low-income districts can’t necessarily counteract the out-of-school environment, but at least it’s not moving in the wrong direction.
September 10th, 2009 at 9:50 am
The total measurement of government worker productivity was shut down by the BLS after the Reagan/Bush regime when total government productivity dropped by half. This article in the fedgazzette reviews the history of measuring government worker productivity.
My interpretation of the facts is that Ronald Reagan was a damned Communist who wasted trillions in government expansion, then committed accounting fraud to hide the deed. Bush just continued the Communist conspiracy. Where is McCarthy when we need him?
September 10th, 2009 at 9:54 am
But pretty good tests for basic reading and math are available
Good tests for arithmetic exist. Math (particularly mathematical reasoning), not so much.
September 10th, 2009 at 9:58 am
I can’t read the paper itself because of the cost, so perhaps Matt could post actual quotes from the paper itself to back up his claims rather than merely quoting the abstract.
Several points come immediately to mind, though. First, this looks like nothing more than teaching to the tests. Second, the gains, if any, were decidedly modest – 0.28 of a standard deviation translates into maybe a four percent difference. If that. So even if the four percent is a real gain, it’s not something that’s going to be making much of a difference at a ‘failing’ school. Next, the abstract says that there were gains on the conceptual part of the test, suggesting that there were real improvements(and incidentally admitting that teaching to the test is exactly what was going on), but fails to cite the figures. This suggests to me that whatever the gains were, they weren’t even as much as 0.16 of a standard deviation, iow, maybe statistically significant, but not significant. Finally, this seems to raise some red flags:
Hmmm, you don’t suppose that maybe it means something else, say that something that had nothing whatsoever to do with the actual incentive program was responsible for raising the scores?
Now, I don’t know what the paper itself says, and maybe these concerns are addressed in the actual report. But from what’s been posted so far, Matt has come up with a big fat goose egg. And this is the best cite he can come up with in support of his position.
September 10th, 2009 at 10:07 am
My wife works in public (middle) school education in MA. Teacher for 8 years (was union VP for 3 of those, and led negotiation for the union on a 3-year contract), vice principal for 3, and now in her 2nd year as principal. I asked her last night what she would change to improve teacher quality. In order of importance, here’s what she said:
1. eliminate tenure.
2. eliminate accreditation.
3. individual rather than collective bargaining on salary.
On (3), she thought it much more important for the sake of acquiring talent, not retaining it.
She also said that if she were at a high school, she might bump up individual bargaining to #2, as the available talent for advanced coursework at that level is harder to come by.
September 10th, 2009 at 10:24 am
What is this obsession with elimination of tenure? Why do you want the teachers to be like assembly line workers?
And I would not trust Indian statistics, especially of we are talking about low numbers like .28 and .16. This may be due to inherent incentives to fuzz the results.
September 10th, 2009 at 10:33 am
Skeptonomist makes a good point, but the opposite may also be true — that there’s more room for improvement (in percentage terms) when you’re starting at the bottom. In NYC, the schools get a “report card,” and a primary measure is how much certain metrics have increased from the prior year. Last year, my son’s excellent upper west side school dropped from an A to B+ (or something along those lines) because it had smaller gains, but that was largely because the scores were fairly high to begin with, so it becomes harder to make significant progress. (Kind of like how it’s easier to lose the first 10% of your weight when you’re dieting, but it gets harder the closer you get to your goal.)
September 10th, 2009 at 10:33 am
More soak the teachers rhetoric from Matt. Looks like everyone’s exhausted from yesterday’s massive thread.
Matt, I would encourage you to look at other ways to improve education other than changes in teacher compensation. That’s probably the area where you get the least return on any reform. Classroom paint color would probably give you a wider swing. Even better–making sure kids get a couple of weeks of test prep and get a good dinner and a good nights sleep the night before the test and a good breakfast the day of. That is, if what we want to do is improve test scores and not just beat up on teachers.
September 10th, 2009 at 10:40 am
> In order of importance, here’s what she said:
>
> 1. eliminate tenure.
I am sorry, but there are very, very, very FEW K-12 school districts in the United States that have a concept of “tenure”. Even the vast majority of community colleges in the US eliminated tenure during the 1970s, and Top 100 universities are working on getting rid of it as well. It just doesn’t exist in any significant amount outside a few northeastern urban districts.
Cranky
September 10th, 2009 at 10:42 am
> 3. individual rather than collective
> bargaining on salary.
Um, individual bargaining is when you sit in a closed office with a manager and say, “Based on my achievements, both quantitative and qualitative, over the past year I deserve and require a $10,000 raise”, he replies “Whereas I think you deserve a $5,000 pay cut”, and the next Friday you find your paycheck is reduced by $5,000/52.
Cranky
September 10th, 2009 at 10:45 am
I dunno… Hyderabad is a pretty muslimy place…
September 10th, 2009 at 10:53 am
I am also puzzled by tenure being a big deal. Don’t most government workers have civil service protection, i.e., they can’t be fired without cause?
September 10th, 2009 at 10:59 am
> I am also puzzled by tenure being a big deal. Don’t
> most government workers have civil service protection,
> i.e., they can’t be fired without cause?
In my experience, school districts tend to be fairly heavy on policies and procedures, as well as an underlying theme of fairness (since that is what they teach to the kids, whether or not they actually believe it being another question), so it is a more laborious and process-driven, well, process to fire someone. Then again, having worked in corporate-land through the Roaring 90s when there were no convenient recessions to blame for “clearing out the deadwood” (as MY so lovingly puts it), it often isn’t easy to fire people in the corporate world either.
But “tenure” per se? I would really like to see some hard stats on where it exists. And whether or not it actually provides any job protection: there is a district near ours that claims to grant “tenure” but that simply means that the teachers who hold it will be last on the layoff list and does not provide any additional job projection (and specifically does not provide the kind of free speech protection that universities claim tenured professors get).
Cranky
September 10th, 2009 at 11:50 am
Re: I dunno… Hyderabad is a pretty muslimy place…
Large Muslim minority, traditional center of Indian Muslim culture and the Urdu language (hence the Urdu name of the city) and ruled by a Muslim king until 1948. But no, it has a Hindu majority, both the city and the former State of Hyderabad.
For the record, while there are no doubt some excellent schoolteachers in India, the worst schools in India are truly dysfunctional, and would make East St. Louis High look like Sidwell Friends. I mean as in teachers not showing up to work on a regular basis, that sort of thing. Which is to say, something like this might be more necessary, and more workable, in India than here. Not that it wouldn’t be useful here as well.
September 10th, 2009 at 12:26 pm
I thought this line from last night’s speech was great speechwriting
Of course it’s not great speechwriting if it causes people to stop concentrating on the speech and to start concentrating on the veracity of the claim…
September 10th, 2009 at 12:27 pm
I always wondered how in the world people posted into the wrong thread. Now I see it’s the site, at least in that case.
September 10th, 2009 at 12:46 pm
Tenure primarily just protects the very worst teachers. Teachers who if parents knew the things administrators did, would cause them to scream bloody murder to their school boards.
September 10th, 2009 at 12:57 pm
“Tenure” is for the protection of the district. Every classroom has one or two sets of parents who disagree violently with the teacher (and, possibly, everyone else in their life). So the district takes the position that they can’t just fire people because you, or somebody else, happen to dislike them.
As for the idea that parents don’t scream bloody murder to the school boards, well, I guess Mike doesn’t pay much attention to his local community.
September 10th, 2009 at 1:08 pm
Cranky,
You take issue with my use of “tenure” and “individual bargaining”.
Your qualms with “IB” seem to be substantive, not semantic. But if you take issue with my choice of words, I just thought it was the natural contrast with “collective bargaining”. No rhetorical shading is intended. I’m happy to refer to them as X and Y.
By “tenure” I use it the way my wife does, to refer to whatever it is that teachers in their first 3 years at a school in her district do not have, and what they do have after that. I seriously doubt it’s the canonical “tenure” of university professors from way back when. Same as above: no rhetoric intended, happy to use another term. But whatever it is, it’s much harder to be fired/non-renewed (and for performance-related reasons, not just budget-related reasons).
September 10th, 2009 at 1:18 pm
The incentive program was implemented in rural schools. The rural parts of India are most similar to the American inner city — high unemployment, broken families, crime, illness, etc.
One thing that may be different between rural Indians and inner-city Americans is that scholastic achievement is not stigmatized in India.
September 10th, 2009 at 1:19 pm
> By “tenure” I use it the way my wife does, to
> refer to whatever it is that teachers in their
> first 3 years at a school in her district do
> not have, and what they do have after that. I
> seriously doubt it’s the canonical “tenure”
> of university professors from way back when.
> Same as above: no rhetoric intended, happy to
> use another term. But whatever it is, it’s much
> harder to be fired/non-renewed (and for
> performance-related reasons, not just
> budget-related reasons).
The corporate world is being held out as that standard here, and other than some extremely aggressive high-tech corporations and some very brutal family-owned firms, it is pretty hard to fire/get fired in the corporate world after surviving the first year.
But I admit to a bit of surprise: I thought supremely competent school administrators were capable of unerringly separating the wheat from the chaff, the good from the bad, and punishing/firing those found wanting. Do you mean that somehow large numbers of probationary teachers are managing to fool these administrators for 3 full years, then go bad sometime in year 4-5? Hmmm, that’s a bit of a problem for the theory of administrator omnipotence.
Cranky
September 10th, 2009 at 1:20 pm
Teaching to the test has had terrible consequences already; witness the wholesale elimination of “useless” subjects like art and music. Add in what has happened under “No Child Left Behind”, with standardized tests diluted or distorted and curricula shaped completely around answering the test questions.
Let’s see. Will you reward people who get high-scoring students and have their scores stay high? Special ed? How do you factor in class size? This is a real-world example of how a seeming benefit can actually backfire. We could end up strongly discouraging innovative teaching and favoring some classes of students over others quite strongly, and all with the best of intentions. The fault is assuming that teachers are the only variable controlling school performance, and that things like out-of-school support, facilities, and class size are not crucial. I’ll buy at some level average performance across large groups as a diagnostic. But your experience with a class of 25 kids can vary wildly from year to year and there are tons of opportunities to game the system.
September 10th, 2009 at 1:30 pm
“The rural parts of India are most similar to the American inner city — high unemployment, broken families, crime, illness, etc.”
This is a too broad and sweeping statement. India is big on family values. In fact many generations stay together. Villages also have less crime than in the slums of Mumbai. Poor, yes (in comparison to the West). But you will find many villagers quite happy with what they have and are willing to share what they have with you. You may even see Dish antennas on the single room dwellings, people with cell phones etc. Illness, yes, but this may be worse in some congested cities like Calcutta or Mumbai. It is actually pleasant to go and visit villages in India. You may be surprised by the hospitality and friendliness in most places.
September 10th, 2009 at 1:56 pm
26: Cranky, I don’t think it’s too much to expect that if you grant job protection after 3 years, there will be some people that do their job well enough for 3 years and then start mailing it in. In the corporate world this happens too. But maybe a little less because a) those people are sometimes fired and b) fewer people push their luck when they know they can be fired. I’m not trying to stake out a strong claim here, just pointing out that the possibility of being fired can be effective even if it is rarely used.
September 10th, 2009 at 2:59 pm
For what its worth, here is a link to page describing how the teacher incentive program would work: TAP
The bonuses would be awarded based on:
50 percent teaching evaluations
20 percent school wide achievement growth
30 percent individual classroom achievement growth
The amount of money we are talking about here is around 2,000 dollars per person.
The bulk of the money going to teachers under this program would be to those who take positions positions of greater responsibility, called master teachers.
As you can see, the weight given to the ability of an individual teacher to raise their student’s test scores is small. The amount of money at stake, while not all that small, is still only 0.05 percent of a 40,000 dollar salary. I could not find out whether or not the bonuses would replace, or be added on to, seniority based pay raises.
As for the study linked above, since I do not have access to the full text, I can not really comment on it. I will note, however, that the gains reported were small, and the study
only ran for two years. Similar results, and length of time, were measured in the only study in the U.S. to replicate these results, the study following the TAP pilot program in Chicago. I would hypothesize that there is a “ceiling” to the amount test scores can be raised using only this approach.
It still feels like we are nibbling around the edges of this issue, and doing it using a framing that is putative to the very people who will be on the front lines carrying it out.
I would not feel comfortable referring to anyone as “dead wood”.
September 10th, 2009 at 3:00 pm
Did the non-bonus-based schools get the same *total* amount of money as the bonus-based schools?
The true test would be to compare a school where every teacher makes exactly $50k to a school where the *average* salary is $50k but can go up or down based on performance.
September 10th, 2009 at 4:15 pm
The funny thing here would be, that the more we argue with Matt about this, the more he thinks we just don’t get it. And this reinforces his view of himself as someone who understands complex stuff that the common people can’t understand.
This entire ‘Dance of the Teachers’ is intended to distract from and hide the core of the problem- that large businesses do not want to pay taxes in big cities to support schools for black children. Thinking that anyone’s life will be improved by scoring 5% better on a 6th grade arithmetic class is just balderdash.
September 10th, 2009 at 5:22 pm
So, MY gets called out for posting unsubstantiated claims about education policy and the next day we get this? Cool… At least it comes from a peer-reviewed journal. I haven’t had time to read the paper, so I’ll refrain from comment.
I would note that many of the concerns expressed in the comments about “teaching to the test” are certainly applicable to performance incentives tied to standardized assessment. I ran a literacy program built on a scripted curriculum for a few years. It was/is a very successful program in that students receive better grades after going through the program, however we didn’t teach them how to read. We taught them how to take standardized assessments designed to measure literacy.
There is a difference.
September 10th, 2009 at 6:37 pm
No system of trying to rate teachers will work until we are willing to admit that in terms of ability, we are not all created equal. In fact, differences in ability also correlate to differences in class, race, ethnicity, etc., and the causes are not always environmental.
As long as we look at abilities unrealistically, we will have unrealistic expectations and cannot adopt any actual workable educational reform.
September 10th, 2009 at 6:43 pm
@ ScentOfViolets
I concur with how unconvincing the 0.28 sigma improvement by itself is, and the fact that there may be uncontolled variables in the study since there is unaccounted for variation.
However, if scores are continuously improving, rather than rising 4% and returning to a steady state, then a 2% (annualized) relative rate of growth will lead to students doing 20% better than the control group in just ten years. Bending the curve, as you will.
September 10th, 2009 at 11:12 pm
@Cranky
1) It hurts new teachers, because it means that unless administrators are sure, they aren’t going to put in much time developing a teacher who it may be dangerous to keep around too long.
2) Some problems it takes a long time to learn about. Its only over an extended period of time that you keep hearing the same problem stories over and over again until you realize you have a problem teacher.
3) People change. Events in their life can negatively impact their classroom performance. Just because a person is doing well at the 3 year mark doesn’t mean that they arent going to run into life troubles down the road that are going to be dragged into the classroom.
4) New administrators inherit entrenched teachers from their bad predecessors. It negatively impacts the ability for a school to be turned around, because a lot of the problem teachers that took root on the previous watch sabotage attempts to address the schools problems.
Administrators dont get the same protections from teachers, so its not surprising that they dont feel a whole lot of sympathy for the teachers getting “tenure”.