Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The 50 million dollar lie

garyrubinstein.teachforus.org

Post reply on HN
Print view View original

61–70 of 159 posts

Re: The 50 million dollar lie

#61

Post by ef4 »

streptomycin wrote: post #22

I'm surprised at the comments here. Seems that nearly everyone disagrees with the blog? Is Bill Gates such a saint among the HN crowd that we ignore statistics for him? Also, see this blog for some more discussion along the same lines: http://ed2worlds.blogspot.com/2013/01/gates-foundation-waste... Seems pretty clear that Gates' study presents the data in a very misleading manner to make a very weak correlation look…

The blog post doesn't have any statistics to ignore.

Graphs are not statistics. Why doesn't he give us a correlation coefficient on his big blue blob graph?

Re: The 50 million dollar lie

#62

Post by absherwin »

ColinWright wrote: post #56
absherwin wrote: post #44

Earlier quoted context omitted.

I'll bite. I won't discuss my experience in my posts since I don't want to argue from authority but will do so below. But first, I should point out that you're committing a similar sin to that which is alleged in the article. What you really care about is the correctness of an argument. You hypothesize that formal training is an important indicator of correctness. Presumably there's also noise in that scatterplot but…

I agree almost entirely. I'm specifically not asking for people to say "I'm a PhD in statistics, and here's the answer. Accept it because I know better than you." What I'm asking for a a complete and reasoned response, along with some evidence as to how much I should listen to you in the first place. If you have no such evidence then the onus will be on you to make your argument more complete, more coherent, and more…

Reinventing the wheel would be a major problem if our goal is to solve the education problem with this discussion. No one here has done even the basic work I would expect of someone trying to understand teacher evaluation as a solution and compare it to other alternatives.

I would argue that the whole point of HN is to think through arguments in other domains and build intuition by reasoning through problems and arguments. Otherwise, what's the point? No one is going to arrive at this thread and scan the top-rated comments for the solution to his school district's problems.

In cases where actual decisions are being made where the analyses are much more thorough and fully validating much more expensive, other techniques are available. First, one generally builds an awareness of the strengths of each team member which suggests where errors may be more likely. Additionally, one can check a random set of the most likely problem areas. Perhaps, most importantly, while everyone won't re-do every analysis, it's highly unlikely that an any important analysis will only be done once. So one can expect that the high-level results are generally consistent.

Re: The 50 million dollar lie

#63

Post by streptomycin »

ef4 wrote: post #61
streptomycin wrote: post #22

I'm surprised at the comments here. Seems that nearly everyone disagrees with the blog? Is Bill Gates such a saint among the HN crowd that we ignore statistics for him? Also, see this blog for some more discussion along the same lines: http://ed2worlds.blogspot.com/2013/01/gates-foundation-waste... Seems pretty clear that Gates' study presents the data in a very misleading manner to make a very weak correlation look…

The blog post doesn't have any statistics to ignore. Graphs are not statistics. Why doesn't he give us a correlation coefficient on his big blue blob graph?

You criticize a short blog post for something that is not done in the Gates publication. Do you have some bias, perhaps? Why does Gates not even give us a scatter plot, let alone a correlation coefficient? This is the critical point raised by the blog.

Re: The 50 million dollar lie

#64

Post by Alex3917 »

ryanmolden wrote: post #57
Alex3917 wrote: post #8

Earlier quoted context omitted.

"Whereas to me it looks like it removes visual noise to show an actual trend." Except for that in this case the noise is more important than the trend. Think about it, if you're firing or sanctioning perhaps 30%+ of teachers each year for no reason, then only complete morons would go into teaching. It's the same as airport security, where a .1% false positive rate is unacceptable, whereas a 75% false negative rate is…

Did you transpose false negative and false positive in your statement about airport security? I think airport security would much rather have false positive (i.e. This guy has something suspicious, let's do further checks, oh, turns out it was nothing) vs. False negatives (i.e. This guy is clean, oh, turns out he wasn't and blew up a plane).

No, that's correct. The issue is that the overwhelming majority of people aren't terrorists. So even though it's worse to let a terrorist on a plane than it is to ban someone who isn't a terrorist from flying, in aggregate the harms of banning non-terrorists from flying become greater than the harms letting a few terrorists fly even when the false positive rate is very small. Bruce Schneier has a good explanation of this somewhere on his blog. (Actually, this is one of his pet issues, so there are probably dozens of blog posts about this.)

Re: The 50 million dollar lie

#65

Post by carbocation »

I'm working on reproducing the graph with the data available from the NYT, but I'm not sure on one or two details of the author's method. If someone sees it, please let me know! Things I can't tell so far:

1) How is a "teacher" defined? Just(hash firstname lastname)? If so...

2) When a teacher teaches multiple subjects per year, which one is chosen? Or are they both represented, so that "teacher" actually means (hash firstname lastname subject)?

Re: The 50 million dollar lie

#66

Post by carbocation »

thedufer wrote: post #43

I'm confused as to why the "‘raw’ score (for value added, this is a number between -1 and +1)" goes up to ~1.6 in 2009-2010 and up to ~1.1 in 2008-2009. Why does the data go outside the ranges he states? I find it hard to believe someone who can't read something so simple off his own graph.

The author incorrectly stated that "value added" is a score between -1 and +1. In fact, "value added" scores are z scores, which can in theory range from (-∞,∞).

Re: The 50 million dollar lie

#67

Post by Evbn »

FourthProtocol wrote: post #5
kenjackson wrote: post #2

This is a surprising blog post in that I draw the complete opposite conclusion the author does. The author seems to think that the averaging hides volatility (which it does), which leads to incorrect conclusions drawn. Whereas to me it looks like it removes visual noise to show an actual trend. In his original scatter plot, because of the big ball in the middle it's easy to handwave and say, "look a random blob" -- b…

The other thing the author does is to derive absolute findings when the author himself admits to making an assumption about how the numbers were obtained. To me, using the word "lie" in the title is therefore nothing more than click-baiting. And on a separate note - there's a lot of anti-Gates sentiment flying around lately. Whether his methods are the best or not, he's using his own money . I find it really difficul…

Many would argue that Bill came by that money via illegal and unethical behavior, paid for by millions of consumers, so it is more like tax money than his own money.

Re: The 50 million dollar lie

#68

Post by Evbn »

FourthProtocol wrote: post #5
kenjackson wrote: post #2

This is a surprising blog post in that I draw the complete opposite conclusion the author does. The author seems to think that the averaging hides volatility (which it does), which leads to incorrect conclusions drawn. Whereas to me it looks like it removes visual noise to show an actual trend. In his original scatter plot, because of the big ball in the middle it's easy to handwave and say, "look a random blob" -- b…

The other thing the author does is to derive absolute findings when the author himself admits to making an assumption about how the numbers were obtained. To me, using the word "lie" in the title is therefore nothing more than click-baiting. And on a separate note - there's a lot of anti-Gates sentiment flying around lately. Whether his methods are the best or not, he's using his own money . I find it really difficul…

Luxury taxes crimped the yacht industry and many workers. Fiscal conservatives would appreciate some support for the yacht industry.

Re: The 50 million dollar lie

#69

Post by spikels »

vy8vWJlco wrote: post #36
feral wrote: post #7

I know nothing about the domain of teacher measurement, and have no opinion on it. But the first chart in the blog post, with the big mass of blue on it, is surely not the strong evidence of weak correlation, that the author is making it out to be? There could still be a strong correlation in that data, even if it looks like a blob of blue - because there are so many data points on that chart, that we can no longer t…

A heatmap would be more revealing, but I'd eat my hat if it weren't a standard distribution (and I had a hat). It's nature (but hey, anything's possible). The scales are only slightly skewed, but we're not talking logarithms. The merit of the scales ("value-added") on the other hand, is very questionable. But the relation, whatever it is describing, is pretty weak since the ellipse isn't very irregular (it's only "sl…

Clearly all the overlapping points in the first graph are obscuring most of what's going on. This is a warning flag that the author may be trying color the facts to suit their argument.

Not very clear what you are try to say:

(1) What "standard deviation"? Perhaps you mean the standard error of the correlation coefficient is large relative to the its estimated value. Given the large number of data points it is likely to be quite small. Another warning flag is the failure to report either the correlation coefficient or its standard error.

(2) What does "ellipse isn't irregular" mean? Given all the overlapping data points the shape of the plot is entirely driven outliers. To my eye this looks what you would get from plotting a bivariate normal distribution.

(3) Kepler? The linked graph? How is this helpful in understanding what's going on?

The first graph has a straightforward interpretation: you are making two imperfect measurements of each teacher's performance at two different times. There is noise in both measurements and teacher performance may have actually changed between measurements.

Re: The 50 million dollar lie

#70

Post by Evbn »

ColinWright wrote: post #9

I suspect there'll be a lot of differing opinions on this, and I'm looking forward to seeing the discussion. What would be helpful would be if people could say how much experience they have in hard statistics, and how much what they say is driven a priori from the data. I know that hackers, in particular, have real problems with "Argument from Authority", but stats is one place where it's really, really easy to go wr…

Or you could ask people to put forth coherent mathematical arguments, since research has shown, for example, that most professional PhD-holding published medical research is statistically incorrect.
Post reply on HN