The 50 million dollar lie

This is a surprising blog post in that I draw the complete opposite conclusion the author does. The author seems to think that the averaging hides volatility (which it does), which leads to incorrect conclusions drawn. Whereas to me it looks like it removes visual noise to show an actual trend.

In his original scatter plot, because of the big ball in the middle it's easy to handwave and say, "look a random blob" -- but even a slightly longer look seems to indicate there is a positive correlation in the data. His averaging that he does at the end, IMO, makes it clear.

You're missing the point. The author agreed that there is a slight positive correlation in the data. But the averaging makes it look like there is a strong positive correlation in the data, which there isn't.

In other words: without the averaging, you are seeing the full implications of the data: slight positive correlation, but a lot of scatter--i.e., not much meat there. With the averaging, you are seeing only the positive correlation, with all other information filtered out--i.e., you are seeing only part of the picture, and it's the part that, surprise, surprise, makes the person who ordered the data collected look good.

+1.

The blog author saying there is no clear trend in his random blob of points is misleading. The trend is there, it is just difficult to tell the density of points in his scatter plot because of the marker size and plot size.

"Whereas to me it looks like it removes visual noise to show an actual trend."

Except for that in this case the noise is more important than the trend. Think about it, if you're firing or sanctioning perhaps 30%+ of teachers each year for no reason, then only complete morons would go into teaching.

It's the same as airport security, where a .1% false positive rate is unacceptable, whereas a 75% false negative rate is just fine.

I see the OP's point, that visual noise can be filtered in such a way as to create a more compelling trend, and perhaps the filtering that Gates has done has simplified the results...so, OK. Let's not look at charts, let's look at actual calculations that have the various factors accounted for...what are they?

And I guess I'll throw in my two cents into the inevitable teachers-vs-data debate. I believe teachers may be too quick to toss aside performance analysis because they are so close to the personal stories and circumstances...no line chart can seem to account for the student who was improving greatly but then missed an insurmountable number of classtime due to problems at home.

Most humans are unwilling/unable to see their lives and work measured as averages and trend lines and deterministic outcomes, and I think teachers are acutely of this belief due to the nature of their work. And unfortunately, this blinds them to the possibility that while there is no cookie-cutter solution that always works, there are surely some measures/guidelines that often work, or at least mitigate even the most extreme circumstances. As it is, there's not much incentive or opportunity to see "the bigger picture" outside of what happens in one's own classroom year after year.

That said, this is not even close to being mainly the fault of the teachers. Among the other problems is that administrators are just as prone to shortsightedness, and "the numbers" can be wielded by a petty/mediocre principal/district officer to apply pressure to a teacher. It's no wonder that teachers are quick to distrust analyses, given the possibility that they're abused for political means.

So, essentially, it's a giant crapfest with no easy solutions. From the perspective of Gates' efforts, we can only hope that he pours in so much money and advocacy that there's eventually a shift towards best practices...though it's safe to say there will still be unintended consequences that aren't all favorable.