Performance Bonuses
I think there are definitely some obvious problems with Texas A&M’s plan to offer cash bonuses to professors who get good student evaluation scores. Most notably, you could imagine a lot of profs gaming the system. But a lot of the very hostile reaction to this idea among academics with blogs strikes me as less about the flaws with the process than about the inevitable human tendency to want to avoid being held accountable to impersonal quantitative metrics. You can see that from the nature of Dave Noon’s counterproposal:
If the administration at A&M were serious about improving classroom performance, they’d invest quite a bit more money in pedagogical training for their graduate students; hiring more professors and reducing class sizes; offering release-time for professors to design new courses; and so on and so forth. But since they’re clearly not serious, this is what they’re offering instead.
These are fine ideas, but they’re talking right past the problem A&M is trying to address, namely that professors at large research-oriented universities have little incentive to focus on the quality of undergraduate education. These are all proposals that would assist a properly motivated professor to improve the quality of his or her instruction. But the do little to change the behavior of either the jaded older faculty member or the harried junior faculty member. Adding a direct financial payoff to effective instruction is a perfectly reasonable idea. But you need a much better measure of “effective instruction.”
For some courses, I’m not sure how you do that. I took a small seminar on Vichy France in college and it seems to me that it would be hard to develop a metric of how well Professor Higonnet did with that—though if you ask me he did extremely well. But I also took a lecture course on optics. The professor was extremely dull and the TA had a questionable command of English. Consequently, both got poor evaluations from me. Also consequently, I missed many more lectures from this class than from any other. And my sections with the TA weren’t very helpful. So my scores on problem sets and exams were consistently bad. And for my meager efforts I was rewarded with not-so-hot grades.
Which is fine. The grading process was an imperfect assessment of my understanding of the subject, but it wasn’t a totally useless one. I clearly deserved a worse grade than some other people in that class, and also deserved a worse grade in that class than I got in some other courses. I got what I deserved. By the same token, if we compare two low-level science lecture courses, and see that one course has consistently higher attendance and students get higher scores on their problem sets and tests and the professor and TAs are getting higher evaluation scores then there are really only two things we can conclude. Maybe one class is offering better instruction, in which case it makes sense to reward the instructors and do something to bring about improvement in the other class. Or maybe one class is offering inappropriately easy tests and problem sets and thereby gaining praise from students without really helping them learn, in which case the curriculum should be made more rigorous. I don’t think it’s obvious exactly what the best way to do this would be, or how you would apply it to all different kinds of coursework. But there are definitely steps in this direction that can and should be taken at the higher education level, just as they should in K-12 education.
52 Responses to “Performance Bonuses”
Register
RSSLogin
Topic Cloud
ArchivesBlog RollAbout Matt YglesiasContact Me
Donate
January 14th, 2009 at 1:53 pm
Higonnet may or may not have been a good professor, but his book on the rights of nobles during the French Revolution was one of the most interesting FR books I’ve ever read.
January 14th, 2009 at 1:53 pm
“These are fine ideas, but they’re talking right past the problem A&M is trying to address, namely that professors at large research-oriented universities have little incentive to focus on the quality of undergraduate education.”
This is exactly right, and it’s probably the single biggest problem in our system of higher education today.
As a poorly-paid TA who received good student evaluation scores, I certainly wouldn’t have complained about cash bonuses. But such a system is eminently corruptible. If nothing else, it offers yet another perverse incentive for grade inflation.
Making these evaluations a larger part of a faculty member’s tenure review, however, would probably be appropriate. Unfortunately, given the extent to which the sciences are driven by research funding, I think it’s unrealistic to expect that large research institutions will ever pay more than lip service to undergraduate education.
January 14th, 2009 at 1:53 pm
I think it has more to do with wanting to avoid being held accountable to perversely-incentivising metrics that are (at best) tangentially related to what the job they do actually entails.
January 14th, 2009 at 1:57 pm
First of all, many big universities do consider evaluations (and enrollments) alongside research when giving merit raises. I know mine does. But those annual increases are always peanuts compared to the bump one gets by obtaining an outside offer, and those are driven by scholarship.
Second, the emphasis on evaluations tends to reward classes that require less work and offer easy grading. There are things departments can do to adjust, but emphasizing evaluations naturally encourages grade inflation.
That said, I do favor assessment, if only to know which courses and instructors are truly terrible.
January 14th, 2009 at 2:00 pm
There is no need for students to make these assessments. The University could hire an outside auditor to measure the quality of a class. I don’t think that it is very hard to make a good assessment of which classes are well taught, the disparity between good and bad teachers at the college level is fairly wide.
January 14th, 2009 at 2:05 pm
@AWC: Hey, if you want assessment of that type–which classes and people to avoid–we already have Rate My Professor. Not that will help universities, but it’s a godsend to students. Though there are of course times where “terrible” is conflated with “difficult” on RMP, but usually you can tell one from the other by reading the reviews.
January 14th, 2009 at 2:07 pm
“These are fine ideas, but they’re talking right past the problem A&M is trying to address…”
Well, that’s because A&M is addressing the wrong problem. Here’s what an A&M administrator said:
“But the fundamental purpose of this program, she said, is to measure student satisfaction, not teaching effectiveness.”
So, yes, if you want to measure how satisfied students are, evaluations are all you need. I’m not sure if that’s worth ten grand, though, since a university doesn’t exist to please students but to educate them. How nice it is when those go together, but they don’t always and they shouldn’t be expected to.
January 14th, 2009 at 2:09 pm
There are two issues at work here.
The first one you have already mentioned – At a recent examination at the college I work at, final grade for the class and teaching evals were correlated at +0.67. So, as you mentioned, make your class easier=better evals.
The second is some classes are just not enjoyable. I teach in psychology and all majors have to take a stats class. I have seen many iterations of this class. Some instructors can make it somewhat less painful than others but it is just inherently unpleasant material at the novice level.
This unpleasantness of the material is frequently attributed to the instructor, who will receive (relatively) poor evaluations.
Instituting merit raises based on teaching evaluations would result in profs avoiding these classes more than they already do.
These would then be farmed out to adjuncts who typically do a poorer job than the tenured or tenure track faculty.
Teaching evals definitely have merit but that merit falls largely in situations where profs use student feedback to improve their classes.
January 14th, 2009 at 2:11 pm
You’ll also note that the professor’s solution to the problem also works if the problem were that professors work too hard. Funny how ‘education professionals’ solutions to various problems always seem to involve solving the professors work too hard problem simultaneously.
January 14th, 2009 at 2:11 pm
Would you have patients review doctors?
January 14th, 2009 at 2:15 pm
I’d say a lot of the very hostile reaction to the idea among academics with blogs is about
a). The extensive research that most academics who care about teaching know about, which shows that course evaluations are not just “impersonal quantitative metrics,” but in fact highly problematic data that tend to conform to race and gender biases and tend to reward easy grading and light workloads.
b.) The fact that Ezra slammed academics broadly and sweepingly for not giving a shit about teaching, when many many academics, including those who bitch about undergrads, find teaching to be the most rewarding and important part of their jobs, a combined emotional and intellectual challenge without parallel.
c.) The fact that academics often reserve a special kind of hatred for the student, usually bearing an uncanny resemblance to Messrs. Klein and Yglesias, who is quite smart but often overestimates the reaches of his knowledge and intellect. Part of this is simple self-loathing on the part of academics who were themselves the exact same kind of undergrad. Another part is that sometimes you little twerps need to get knocked down a peg or two.
January 14th, 2009 at 2:17 pm
I’ve taught at several different kinds of institutions — small colleges, and now a Big-10 research institution — and I think Matt is right about the incentive issue. At big research universities, paying attention to undergraduate education does little to advance your career.
It’s important for people to grasp that this isn’t true across the board. At a small New England college where I once taught, teaching evaluations were tremendously important. Probably too important, actually — it tended to produce a culture of student-flattery.
At research universities, I think it would be a good idea to pay closer attention to evals, though I would strongly recommend that this be coupled with a system that fights grade inflation — e.g., by considering relative rank within a class more than absolute letter grades when GPA is being calculated.
January 14th, 2009 at 2:17 pm
Charlie: Well said. I’d substitute “frustration” for “hatred,” though, at least in my case–teachers care about their smart students, which makes it more aggravating when the limits of their thinking are exposed.
January 14th, 2009 at 2:17 pm
The A&M administrator has this exactly right, student evals are a measure of “customer satisfaction.” To the extent that universities are ranked and ultimately garner alumni donations partly on the basis of how “satisfied” students are with their experience there, tracking this makes sense. But all the research on this shows that the biggest drivers of student ratings are (1) expected grade and (2) personal affinity for the instructor. A system that rewards student ratings (every university has one to a greater or lesser extent) encourages grade inflation and general pandering to students.
January 14th, 2009 at 2:25 pm
tomemos, thanks. Make that “irrational flashes of frustration” instead of “hatred.” Of course we love the little smarties. In fact, the sheer enjoyment of an apt pupil is what leads some academic to try to land jobs at those research 1 schools, where they can teach grad students.
January 14th, 2009 at 2:25 pm
A number of good points have already been made by the commentators. I’ll just add that when I taught at a Big Ten U, it was a well-known tip that to get higher scores, you should deliberately pander to the metrics. I kept getting a low score for the “professor is well organized” metric, until someone pulled me aside and told me to have a slide that said “Today’s Organization” at the beginning of every class. My score on that metric went up when I started doing that, even though nothing else in the class really changed.
January 14th, 2009 at 2:26 pm
It seems pretty clear that part of the problem that Matt and Ezra are having with understanding the reaction of many academics to this idea is that Matt and Ezra were good students. If I were only evaluated by students who had, say, GPAs of 3.0 or better (I’d love to hold out for 3.5), I’d be a lot more sanguine about prizes or other incentives being attached to student evaluations. Poor students (meaning especially those who don’t want to be challenged intellectually) are much more likely to evaluate their professors on inappropriate criteria, such as the grade they expect to receive.
What it really boils down to is that the absence of good ways to evaluate or incentivise teaching is not a good reason to rely on bad ways.
For a related discussion of incentives in a different field, see Incentive Pay Considered Harmful at Joel on Software.
January 14th, 2009 at 2:32 pm
Mega dittos, as it were, to comments #4, 7 & 10.
As to high correlation between grades and evals, don’t forget the perverse incentive on the other side. When you’ve completed a course that for whatever reason just hasn’t gone well, and you know the evals are going to reflect that, give an absolute bitching final exam so that the grade distribution will be lower than normal and attribute the poor evals to the students’ lousy grades.
January 14th, 2009 at 2:35 pm
It’s already the case, where I teach, that the committee that assigns my yearly merit raises is instructed to consider my teaching, including my student evals.
But having served on those committees, I can tell you that the evals don’t count for very *much*, and the pool of money used for raises is so small to begin with that the maximum difference made by teaching evals might be fifty bucks a year.
If research universities really wanted to encourage people to pay attention to teaching, they would need to adjust tenure criteria. Right now it’s explicit that you will lose your job if you let teaching get in the way of your research agenda. Compared to that anxiety, the merit pay issue is really small beer.
January 14th, 2009 at 2:36 pm
Uggh. Teaching evaluations are next to useless, and other commenters point out, reward easy graders and panderers.
Probably too important, actually — it tended to produce a culture of student-flattery.
This is too true. I teach a low-level course on logic and argumentation. There is sometimes a certain student in the class who gets very indignant when I tell him that because A leads to B, this does not, in fact, mean that B leads to A. Sometimes this certain student tries to come up with reasons as to why his “logic” is actually correct, at which point I just move on. Now, I could pander more to these people, tell him (it’s almost always a him) that, hey, you have a point, good thought….except he doesn’t and it’s not. I’m not an asshole about these things, but I’m not going to reward ignorance or play to this kid’s ego just so he can feel better about himself.
January 14th, 2009 at 2:36 pm
But there are definitely steps in this direction that can and should be taken at the higher education level, just as they should in K-12 education.
So what are those steps?
You’ve correctly characterized a problem with higher education, and sketched the difficulty of the problem, but ending with “there are definitely steps” is weak sauce.
Here’s one (relatively off-the-cuff) idea: in course evaluations, ask students to rate “how much do you think you learned in this course?” in addition to how much they enjoyed it and/or how difficult it was. Dole out the cash based on the answer to this question.
January 14th, 2009 at 2:38 pm
Actually, that should be #4, 7 & 11 in the above post.
Re #10: While patients clearly lack expertise to judge most of what a doctor does, patient evaluation of doctors might be good at identifying those who don’t keep you waiting for 2 hours, who actually listen to your description of symptoms, and at differentiating between doctors who treat a patient vs. doctors who treat a disease.
January 14th, 2009 at 2:44 pm
Would you have patients review doctors?
As the wife of a doctor I can tell you that there are generally no shortage of metrics available for medical groups to rate their doctors. Medicine is unlike teaching in that respect. It is easy enough to rate doctors based on complex matrices based on how many and what type of patients they’ve seen per time period, how many procedures they’ve performed etc. And most bonus systems in the medical profession are based on actual production. Earn more money for the clinic and you earn more money for yourself.
The problem with teaching is that it is enormously difficult to rate teachers or professors in any sort of fair and systematic way. It’s generally easy to pick out the very best and worst teachers. But much more difficult to sort through those in the middle.
I teach now, but I previously worked for 15 years for a science agency in the Federal government that tried to institute an experimental merit pay system. The whole thing turned into an enormous burden and didn’t really resolve anything. The workload for supervisors increased greatly because they were required to undertake much more extensive performance reviews of their employees once those reviews became the basis of merit pay. And employees were burdened with considerable extra busy work as well to generate all the ongoing documentation upon which merit pay bonuses would be doled out. And in the end it really didn’t make much difference at all because the supervisors basically doled out the merit pay bonuses equally across the board. Top performers got a little bit more and bottom performers got a little bit less. But for the 90% in the middle their lives and pay went on pretty much as before, making the whole exercise pointless.
January 14th, 2009 at 2:48 pm
If the A&M evaluations are similar to the bubble sheet evaluations I’ve filled in (and had filled in about me), then I agree with those commenters who consider student evaluations to be lousy indicators of course quality. 5 point scales on a single sheet to assess a 10-15 week course? That’s many kinds of inadequate.
However, if they’re a more substantive from of evaluation, which I’ve given to students with an eye toward improving courses I’ve taught, then I’m not as bothered by the idea.
January 14th, 2009 at 2:53 pm
My experience is in some ways parallel to Kent’s (#24), but I should also point out that in most universities there is already an enormous structure of reviewing and assessment in place to track the progress of junior faculty toward tenure.
It does take a lot of time, but we already invest the time, because the whole tenure system would be meaningless without rigorous review of qualifications.
The thing is that the qualifications getting assessed are very different at different kinds of institutions. Where I now teach, undergraduate teaching makes very little difference. 5-10%, I would say. Where I used to teach, it was more like 65%.
Honestly, if consumers wanted to create pressure for institutions to attend to undergraduate education, they could do it by voting with their feet — at least they could if this sort of information about the differences between institutions were made more widely accessible.
January 14th, 2009 at 2:58 pm
Design a scoring system that awards points based on alumni donations prorated by the percentage of credit hours each professor taught the donor.
Yeah, it’s stinks, but any gaming of the system results in increased donations to the school!
January 14th, 2009 at 3:05 pm
When I taught I was struck by how closely the class evaluations I got were correlated with the grades the students were getting. This has a pretty perverse effect on teaching quality because giving easy grades early is a good way to guarantee that students do not bother to work in a class. Students who get easy grades up front assume they don’t have to work hard and their work tends to deteriorate as the semester goes on.
To get students to work I tended to grade harder earlier in the class (at least compared with what they could be expected to know at that point, I was teaching philosophy, so part of what I was teaching regardless of the class was writing coherent papers) then at the end. But because evaluations were done before the final grades were in, my students likely expected worse grades than they actually got in many cases.
The strict correlation was not between student evaluations and student grades. It was between student evaluations and student expected grades. (Some of the forms have the students put down their expected grades, so it becomes pretty easy to see how strong the correlation is).
January 14th, 2009 at 3:14 pm
This should be dealt with at the hiring level.
When we interview job candidates, we read their research and fly them out to do a job task. Few research universities pay comparable attention to teaching. Watching candidates teach a class or two (in person or on videotape) would at least make it possible to make this a consideration in hiring decisions, certainly more so than just reading teaching dossiers.
January 14th, 2009 at 3:22 pm
Look, if anyone wants to suggest that us university instructors ought to be held (more) accountatble for objective teaching outcomes–e.g., scores on tests that aren’t dumbed down–then we can start having a real discussion. This might work in somce of the science. I’m really not sure how you’d measure teaching outcomes in essay-based classes. But whatever. There’s a serious debate to have there.
But that debate has nothing to do with Texas A&M’s proposal, which doesn’t even try to measure actual teaching outcomes.
January 14th, 2009 at 3:24 pm
January 14th, 2009 at 3:26 pm
Agree with Ian (#30) that this issue should be addressed in hiring.
Once again let me stress that different institutions are very different in this regard. At many small colleges, the hiring process is already all about teaching — including “sample classes.”
In short, this isn’t a technical problem — where we just don’t *have* any way to assess teaching. Some institutions are already doing it, and doing it rather well. But since research institutions get much of their money from research, they don’t actually *want* to make undergraduate teaching priority #1.
The Texas A & M solution is bad mainly because it’s an attempt to incentivize teaching “on the cheap,” without making any of the tradeoffs you would actually need to make.
January 14th, 2009 at 3:26 pm
The biggest problem I have with giving bonuses based on student evaluations is how poorly the evaluation instruments are designed. My big research university uses a five-point rating system for each question, but it is worded in such a way that a professor has to be really bad to get a two. Functionally the scale is from 3-5. This does not give students a chance to differentiate professors as much as they would like. My rule of thumb was a professor had to get above a 4.5 to be considered really good and below a 3.7 to be considered bad.
As a student I would much prefer to have a rating system of:
1. Really Bad
2. Below Average
3. Average
4. Pretty Good
5. Good
6. Really Good
7. Awesome
The gradations are much finer on the high end because that has been my experience with professors – very few were terrible, most were varying degrees of good. I have classified professors that went from pretty good to really good as good, even though that isn’t fair to the really good profs and is too generous to the pretty good profs.
The other objection I had was to the questions themselves. Course is well-organized, Professor cares about student progress, Professor made me feel free to ask questions. You could get better information by asking students how many times they skipped class because they didn’t want to sit through the lecture, how much value the professor added beyond the textbook and whether they would recommend the professor or the class to any student, students with an interest in the subject or just students in that major.
As too the correlation between grades and a professor’s evaluations, I would be curious to see if the correlations varied by the difficulty of the university. I wonder if the correlation holds at schools with higher expectations placed on students – Texas and Texas A&M versus Texas Tech and Texas State. The professor ratings I have studied at Texas (undergrad and law) do not seem to correlate that highly with workload – a lot of professors with high or excessive workloads still seem to get 4.5 or higher ratings, and a decent number of professors with light workloads get 3.8 or lower ratings.
January 14th, 2009 at 3:45 pm
Wouldn’t it need to be an eight-point scale instead?
1. Really Bad
2. Below Average
3. Average
4. Pretty Good
5. Good
6. Really Good
7. Awesome
8. Awesome and hot!
January 14th, 2009 at 3:56 pm
The interesting thing is how little interest there has been in the topic of improving college teaching as opposed to the vast amount of attention devoted to improving K-12 teaching.
January 14th, 2009 at 4:29 pm
Even this doesn’t work. I read a story by a philosophy professor who was teaching two sections of the same course. He tried an experiment. One course he taught as usual. In the second, he spoke in a higher tone of voice and was more animated on the stage. The second class gave the professor higher marks for knowledge and organization, and said they learned more, even though their grades were the same as the first group.
January 14th, 2009 at 6:06 pm
Not sure they will ever get really serious about this unless research funding is decoupled from promotions. The University tried to get my boss to pay his entire salary out of research funding. Then they take half of every research grant for “overhead.” Finally, if you need something, like using facilities, they charge money.
January 14th, 2009 at 6:20 pm
First, there are lots of incentives to teach well, including, at least everywhere I’ve been, a strong culture of wanting to teach well.
And I totally agree with 3. Really, you and Ezra need to spend some time trying to derive useful information from student evaluations. Personally, I would KILL any small cute animal of your choice for an objective, or remotely reliable, metric by which I could evaluate and steer my teaching. Not just to shut up people like you and Ezra
but because I really want my courses to be genuinely beneficial on a number of fronts.
(Btw, a lot of the things you could more or less reliably derive from evaluations you can just as easily derive from simple observation. If the TA or professor cannot fluently speak the language of the students, there will be problems. If they talk too fast or leave too many lacuna in their explanations…you’ll see it pretty quickly. So why course evals?
I take evals as some sort of highly unreliable evidence. I’m not sure why I still think they are worth thinking about except that I do believe, against most evidence, that they tell me *something* and something that, if I’m lucky, might help me improve a course. But to assert that they are any sort of objective metric *of the instructor* is…well…nuts.)
(I think I’m getting bent out of shape
)
BTW, what *is* the evidence that professors at large research institutions (in which fields?) don’t have incentives to teach well and/or don’t try to teach well?
January 14th, 2009 at 6:44 pm
I agree with you (Bijan) that student evaluations are maddening. I would say they have a correlation — but a pretty weak correlation — with the actual quality of instruction.
And look, of course we all try to teach well. For one thing, it’s pleasant when a class is going well, and it’s a drag when the students hate it or don’t get it.
So we’ve got an incentive, of sorts. But I can tell you that at my institution there aren’t a lot of hard *pressures* to make a course work well. And there are a lot of hard, inflexible *pressures* to make your research agenda productive. So in a world with limited time and energy, when the chips are down and you have to make trade-offs . . . you reduce the amount of time you’re spending on your courses. I don’t think field makes a huge difference. I’m in the humanities, so research money isn’t a direct rationale for this set of priorities, but we absorb the broader culture of the institution.
I suspect this is most true at large, *public*, Carnegie I research institutions. Private research institutions often have more resources, so (at least at the top of the pile) they can sometimes do a better job of having their cake and eating it too.
January 14th, 2009 at 7:10 pm
re: objective, impersonal quantitative metrics, I think Jack Maple is interesting:
http://www.govtech.com/gt/94865
“Q: You advocate accountability for the people in charge of the precincts. Why is that so important?
A: At the weekly COMSTAT meetings, you have the heads from every precinct and division. You put all the intel up on huge maps where everybody can see it, and you start asking them what is going on with the crimes or patterns in their area. You hold them responsible. You ask tough questions. No one is in trouble because there is crime, they are in big trouble if they have no idea what the crime is, they cannot tell you where the crimes are and they don’t have a plan to deal with it.
It is a four-level process, with chiefs and deputy chiefs being held accountable. Then they hold the borough commander accountable and they in turn question the precinct commander. The next level is the precinct supervisors, and finally the sergeant saying, `Maple, tell us about the last 10 robberies on the post. Jones, you think that’s funny? Tell us about the last 10 burglaries on yours.’”
Of course, there are metrics and then there are metrics. # of robberies is a real metric, squiggles filled out by students after finals is not quite the same thing. It would be like paying police officers based on citizen surveys of their perceptions of crime.
January 14th, 2009 at 7:50 pm
A note from the trenches:
If you have high standards, don’t let students drop their lowest grade; don’t allow infinite resubmission, penalize late work and see an A as standing for excellence, teaching evaluations will suffer on average. Factor in some other issues: Timing of class, gender/race/diction, cookies or no cookies on eval day; required class or elective; entry-level or specialized class taught to majors; teaching style (ooh for the Powerpoints with applause buttons embedded in them) — it all leads to too many contradictions. Evals can be useful at the extremes — very few professors walk on water, very few are completely hopeless — looking at distributions of the evals as well as expectations, long term GPAs for the professors (not the students!) also might . We already know that adjuncts whose employment depends on evaluations — one bad set and you’re gone — HAVE to game the system to cobble together enough classes for a job even without benefits….
I’ve observed enough classes (as an evaluator) and taught enough to know that narrowly construed teaching evaluations don’t measure good teaching with precision….
January 14th, 2009 at 9:01 pm
Well, I notice one of the classes Matthew disliked was one in which the TA had a “poor command of English.”
YOu may also be aware of research which suggests that people evaluate different accents differently, considering a very thick German accent much less harshly than a slight Korean accent.
Also, I don’t mind if students evaluate me harshly, but I have no sympathy at all with students who complain about foreign TAs. You are in a university, so be universal. In the wide world you will have to occasionally try talking to people who speak English as a second language – hell, most Koreans, Italians etc. would be so glad as to have the regular option of dealing with people who spoke their languages as a second language. If you cannot make an effort to understand a TA speaking with an accent, then you don’t belong in any university, let alone Harvard.
And yes, this is where I make the comment about trustfund scumbags – how the hell did you get into Harvard?
This is only half a joke.
January 14th, 2009 at 9:10 pm
But a lot of the very hostile reaction to this idea among academics with blogs strikes me as less about the flaws with the process than about the inevitable human tendency to want to avoid being held accountable to impersonal quantitative metrics.
On the contrary, it is the personal nature of these metrics that is objectionable. If somehow the metrics here were impersonal, that is, they had some sort of control to make them reasonably objective, that would garner far less opposition, methinks.
January 15th, 2009 at 12:10 am
I think a lot of commentary on this thread has failed to see that Matt is actually conceding that evals are an imperfect metric.
The whole point of the anecdote about the class he did badly in is that grades are also imperfect. They measure a lot of things besides the student’s performance. But in the aggregate, they do tell us *something* about that performance.
Similarly, evals tell us *something* about the instructor.
January 15th, 2009 at 12:11 am
This won’t work for all subjects, but I believe it would work for sciences where courses are taken in sequence (math, physics, chemistry). Here’s my plan:
All professors/instructors should have to occasionally teach undergraduate sequence courses (which would be good for them anyway: Nobel prize winner Richard Feynmann liked to teach freshman physics, because the very basic questions he got from freshmen got him thinking in new directions.)
Then, say there are 5 instructors teaching 1st semester freshman calculus. Part of their evaluation should be the average of their students’ grades in second semester calculus (assuming an adequate sample size). If the instructor did a really good job communicating freshman calculus, that should show up as exceptionally well prepared students in the second semester.
You could achieve the same effect by co-ordinating the final exams to a common level of difficulty, then comparing the average scores.
January 15th, 2009 at 3:15 am
Chalk me up as a professor at a research institution who generally enjoys teaching and is also somewhat irritated at the widespread presumption that most professors don’t give a damn about their students. The single biggest factor that limits teaching quality is the time you have to prepare, and the reason that is limited is because we have so many other things to do. A monetary bonus for good teaching evaluations won’t motivate me unless it either improves my chances of tenure (and this varies from school to school) or somehow results in more time to do all of the tasks on my plate.
Plus, as so many people here suggest, teaching evaluations are not very good metrics of teacher quality (and I say this as someone who consistently receives awards on the basis of my evals). Rewarding people in whatever way for their performance on evals creates all the wrong incentives.
That said, it seems to me that the biggest problem is the correlation with expected GPA. What if one addressed this by only counting the evaluations of students who got As? I’m just guessing, but this might result in less of race/gender/etc being a factor, because presumably the better students would care more about whether the class taught them or not. Also it would erase the incentive to grade easily.
January 15th, 2009 at 11:23 am
Well, since we might expect a correlation of evaluation score with GPA, we could run a regression that controls for GPA.
Number scores for teacher performance are pretty limited, though, since they’ll likely be filled with bias. It seems best to look at any qualitative statements about teacher performance, since you can usually tell from the depth of thought put into the evaluation how seriously to take it. If you have an evaluation like “pRofessor LA DOUCHE SUX!!!111!!”, you can kind of take that for what it’s worth.
January 15th, 2009 at 6:08 pm
Big10Prof, while there is often much *less* pressure, and less *actual* pressure, there is a fair bit of pressure to teach well in univeristies I’ve taught at. Or, at least, there is a lot of lip service given to teaching well. The mantra of “research, service, and teaching” is usually really “research, unless your research is good and we don’t like you then we’ll invoke service or teaching” and we all know it. Oh well. There are many attempts to mitigate it in a variety of ways (for example, my uni is trying to let people specialize in teaching more, if that’s what they want).
That being said, it’s clear that there are free rider problems in all this. Some of the people who teach crappily because they are big research shots (or think they are) are generally regarded with annoyance by the rest of us (in my experience).
Plus, there are crappy courses to teach. A lot of people do really well teaching certain kinds of courses and quite poorly on others. Big lecture style classes *can* be pretty good, for what they are. But they can also be deadly for everyone with everyone (teacher and students) trying to slide by with the minimal effort possible.
I still don’t see why Yglesias (or Klein for that matter) still think this is a remotely well thought out or reasonable idea, or that simplistic shots (however softened) are a reasonable response. Actually, it reminds me of debating pedegogy with (some) students (”How can you grade me low when this is philosophy, philosophy is opinion, and I gave you my opinion.”). So I do ask, Matt, why do you think your thoughts (above) were likely to be reasonable? Or helpful? Or not wildly off base?
I’m genuinely puzzled. When I was a student, I took evaluations very seriously, but I also was pedegogically oriented. Most students aren’t. Most students don’t experience a wide range of teachers. Students have strong incentives *other* than learning (getting good grades with a minimum of work; being entertained; being comfortable; having their self perception validated or elevated).
This doesn’t mean that studying students is worthless (pedegogically), but that evaluations (which presume the capability to evaluate a fairly complex object) are (both conceptually and empirically) rather suspect.
Also, given the existence of teaching awards, why would another, less sophisticated award standard be a good idea? This is the other part I just don’t understand: Monetary (both direct and indirect) rewards for good teaching via fairly stringent metrics already exist. And yet, the problems with teaching (and with some instructors) exist. Soooo why is this crappy one worth even talking about?
March 14th, 2009 at 5:11 am
It is the coolest site,keep so!
xanax
March 22nd, 2009 at 6:05 am
tramadol
Great site. Good info
March 22nd, 2009 at 10:24 am
buy viagra online
It is the coolest site,keep so!
April 9th, 2009 at 5:23 am
Thanks for the review! viagra
April 16th, 2009 at 10:11 pm
Hello everyone. I’m moving, but don’t worry! [Someone once] told me we’re all on the same planet, so I’ll be okay!
I am from Iraq and also now’m speaking English, tell me right I wrote the following sentence: “The fastest way to search for cheap flights and airline tickets.”
Thank :p Donald.