Edit: since ColinWright explicitly asked us to indicate experience with hard statistics: Check the link in my profile. I work with very messy data. I've also found that the more experience people have with something, the less confident they are about their expertise so I won't claim to be NotWrong, but the statistics here really aren't hard. It's linear regression with data that appears, cursorily, to be gaussian, close-ish to iid, etc. The question isn't "did Gary do the math wrong?" it's more "is he interpreting the data reasonably?"I will make a strong claim: If doctor effectiveness were this stable, and it were the only data I had available, I would choose a doctor on this metric, ceteris paribus.
I found a scatterplot by the author that uses smaller marks so it looks a lot less blob-ish and more heat-map-ish:
http://garyrubinstein.teachforus.org/2012/02/26/analyzing-re...
I'm trying to track down the original data to verify this myself, but a couple of quick points:
There definitely seems to be good evidence of a relationship. It might be "weak" but that's not the same thing as "ineffective". Considering the number of data points, it makes it unlikely that the trend occurred by chance. This plot also makes it clear why the averaging made it look so pretty: the average in just about any percentile group in 08-09 matched 09-10 regardless of how finely it's divided (to a point).
It's certainly the case that there's a lot of "noise". The question is how to interpret it. One might conclude that there is a lot of "measurement noise" -- that teacher evaluations are inherently messy and rather uninformative except as a larger trend. There seems to be reasonable support for this! Students aren't the same, even from year to year. Of course it could "signal noise" - maybe teachers themselves change from year to year. Perhaps one year you're more motivated, the next your not, or vice versa.
To understand the implications of this noise, we also need to ask what we're using it for. For instance, if this were a plot of the correlation between money donated to Watsi and quality-of-life, you'd probably think you were doing pretty well! You might point out that a particular quality-of-life metric is flawed or noisy, or that the quality-of-life depends on lots of things besides health, but likely you'd be happy that you're clearly causing some benefit.
So when we ask whether the data suggest the viability of value-added measurements in aggregate, to me that seems to be a yes. But if we ask whether the data suggest using these measurements to make a decision on a per-teacher basis, that depends on the context of the alternatives! If this is the only thing we have to go by, perhaps it's not so bad.
I would love to see how stable the other parts of the teacher evaluation fit in. If you combine multiple noisy signals, you often get a much better picture of what's going on! This accounts for only 1/3 of the teacher's evaluation. I would like to think that if you appeared not to be a value-added teacher, but your principals and coworkers spoke highly of you, that you'd still get that raise.
Also, I'm looking forward to multiple years of this data - if your score accounts for 35 percent of the variance from one year to the next, multiple years might look a lot better. Perhaps in deciding teacher tenure, scores over 5-10 years could be really useful.