Like all quasi-experiments, this one relies on an assumption that the treatment – here, teacher switching – is as good as random. I find that it is not: Teacher switching is correlated with changes in students’ prior-year scores.
It's important to figure out which kind of classroom we are talking about. There are at least three tiers of classroom styles. If you measure only in the middle tier, then I can believe that teacher skill would have only a small effect. However, it's really easy to tell the difference between the tiers if you look, especially for the bottom-most tier compared to the other ones.
At the bottom tier, some classes are just zoos. The teacher is ignored, and the students talk to each other. At best, they work on material for another class. Teacher skill doesn't matter within this tier, from at academic perspective; one zoo teaches students just as much as another zoo. I am sad to say that classrooms like this do exist. It's a potential bright note that such teachers are very easy to identify in an objective way; their students have absolutely terrible results on standardized tests such as Advanced Placement (AP). There's no need for sophisticated statistics if all the students are scoring 1-2 out of 5.
At the middle tier, some classes involve the teacher walking the students through standardized textbooks and other material. Basically, the textbooks are software and the teachers are the hardware that runs it. It's not an inspiring kind of classroom, but at least it is inexpensive. Within this tier, I could see teacher skill not mattering much, because the students spend all their time glued to the course materials. However, you'd certainly like to find out who is in this tier versus in the zoo tier.
Worth a brief mention is that there's an upper tier as well. Maybe "style" is a better word in this case. Sometimes the teacher actually understands the course material, and so is able to respond to the questions with anecdotes and exercises that are tailored for that particular student. For this tier, teacher evaluation is especially important. Among other things, some teachers are fooling themselves, and would be better off staying closer to the book.
