Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The 50 million dollar lie

garyrubinstein.teachforus.org

Post reply on HN
Print view View original

71–80 of 159 posts

Re: The 50 million dollar lie

#71

Post by Evbn »

colanderman wrote: post #59
WalterBright wrote: post #38

There is no objective way to measure teacher performance. Any evaluation method that can be written as a list of rules can and will quickly be gamed. The thing is, it's easy to find out who the best teachers are. Simply ask the students, parents, and staff. They all know who the good ones and the bad ones are - and that can't be gamed.

Your claim makes no sense. If the evaluation method measures what we want teachers to do, then it, by definition, can't be "gamed": if the measurement is high, then the teacher is doing what we want. Are you claiming that there's something special about teaching that makes teacher performance much more difficult, or even impossible, to quantify? If so, what is the point of teaching if one cannot measure results? Your…

Do you have any such "evaluation method measures what we want teachers to do" in mind? Point is, no one has found one.

Re: The 50 million dollar lie

#72

Post by yummyfajitas »

feral wrote: post #7

I know nothing about the domain of teacher measurement, and have no opinion on it. But the first chart in the blog post, with the big mass of blue on it, is surely not the strong evidence of weak correlation, that the author is making it out to be? There could still be a strong correlation in that data, even if it looks like a blob of blue - because there are so many data points on that chart, that we can no longer t…

I'm really surprised to see Gary Rubinstein continuing to post bad graphs which obscure the data. 8 months ago I wrote a blog post about why you should avoid scatterplots, and used his graphs as examples of what not to do.

http://www.chrisstucchio.com/blog/2012/dont_use_scatterplots...

He is certainly aware of my post - he even commented on it and I offered him some constructive suggestions (such as adding alpha if he wants to use scatterplots).

I'm beginning to think that maybe his goal is to make VAM look bad, rather than to actually clarify and explain the data.

(I also left a comment on his blog expressing my concerns with his plot. Strangely, I seem to be hellbanned on his blog - the post is invisible when I view the post in an incognito window.)

Re: The 50 million dollar lie

#73

Post by Evbn »

ColinWright wrote: post #17
keithwarren wrote: post #13

Typically when someone posts a baiting title like that I know not to trust them. Scholarship and evidence don't need hyperbolic headlines.

I think you are mistaken. In today's attention economy even the very best scholarship and the very best evidence-based conclusions will be completely ignored unless there is a hook to get people, preferably lots of people, to read it. Even here on the allegedly rational Hacker News I've seen item after item sink without trace, whereas others with significantly fewer details and significantly worse evidence get voted…

When is HN alleged to be rational?

Re: The 50 million dollar lie

#74

Post by lancewiggs »

graeham wrote: post #15

There are two problems with this article. First is that the "evidence" is from a scatter plot graph, with no statistical measurements attached to it. Stats really aren't something that you can rely on your eyes on. To me, it looks like a weak, positive correlation between previous year's score and current years score. The thing to look for is greater densities in the NE and SW quadrants than the NW and SE. Having a z…

Exactly. Charts are useful for eyeballing, but only the use of proper statistical tools is valid for showing trends. Charts can easily be very misleading.

There seem to be a lot of comments about the charts on this page. Arguably basic statistics should be a part of every education - is that happening in the USA?

Re: The 50 million dollar lie

#75

Post by jhales »

ef4 wrote: post #61
streptomycin wrote: post #22

I'm surprised at the comments here. Seems that nearly everyone disagrees with the blog? Is Bill Gates such a saint among the HN crowd that we ignore statistics for him? Also, see this blog for some more discussion along the same lines: http://ed2worlds.blogspot.com/2013/01/gates-foundation-waste... Seems pretty clear that Gates' study presents the data in a very misleading manner to make a very weak correlation look…

The blog post doesn't have any statistics to ignore. Graphs are not statistics. Why doesn't he give us a correlation coefficient on his big blue blob graph?

Was thinking the same thing. I think it is: "...correlation coefficient of .35..."

Found it following the link: http://garyrubinstein.teachforus.org/2012/02/26/analyzing-re...

Re: The 50 million dollar lie

#76

Post by ColinWright »

colanderman wrote: post #59
WalterBright wrote: post #38

There is no objective way to measure teacher performance. Any evaluation method that can be written as a list of rules can and will quickly be gamed. The thing is, it's easy to find out who the best teachers are. Simply ask the students, parents, and staff. They all know who the good ones and the bad ones are - and that can't be gamed.

Your claim makes no sense. If the evaluation method measures what we want teachers to do, then it, by definition, can't be "gamed": if the measurement is high, then the teacher is doing what we want. Are you claiming that there's something special about teaching that makes teacher performance much more difficult, or even impossible, to quantify? If so, what is the point of teaching if one cannot measure results? Your…

I'd like to ask - how do you measure programmer productivity?

Re: The 50 million dollar lie

#77

Post by DigitalJack »

Evbn wrote: post #70
ColinWright wrote: post #9

I suspect there'll be a lot of differing opinions on this, and I'm looking forward to seeing the discussion. What would be helpful would be if people could say how much experience they have in hard statistics, and how much what they say is driven a priori from the data. I know that hackers, in particular, have real problems with "Argument from Authority", but stats is one place where it's really, really easy to go wr…

Or you could ask people to put forth coherent mathematical arguments, since research has shown, for example, that most professional PhD-holding published medical research is statistically incorrect.

Could you put forth a coherent mathematical argument for your statement?

Re: The 50 million dollar lie

#78

Post by streptomycin »

absherwin wrote: post #55
streptomycin wrote: post #49

Earlier quoted context omitted.

To provide a contrary hypothetical... Imagine if one group had 55% of its dice labeled up to 110 and 45% up to 100, and the other group had the opposite. So if all you use for hiring is the group membership, you're going to very arbitrarily miss out on a lot of 110s and hire a lot of 100s. And now also imagine that this process of identifying who is in what group is ridiculously expensive.

It's not contrary at all. It's the same point. The changing mix is merely a difference of degree. The key argument you raise is that discrimination has a cost. I don't have the numbers offhand to determine whether the cost of monitoring exceeds the benefit of higher quality education. Regardless, that's a different argument that should be backed by a different set of data.

It's contrary in that a naive interpretation leads to the opposite conclusion. That's all I meant. And I'm not sure how you define what is a different argument and what isn't. It's all about the value of these scores.

Re: The 50 million dollar lie

#79

Post by vy8vWJlco »

spikels wrote: post #69
vy8vWJlco wrote: post #36

Earlier quoted context omitted.

A heatmap would be more revealing, but I'd eat my hat if it weren't a standard distribution (and I had a hat). It's nature (but hey, anything's possible). The scales are only slightly skewed, but we're not talking logarithms. The merit of the scales ("value-added") on the other hand, is very questionable. But the relation, whatever it is describing, is pretty weak since the ellipse isn't very irregular (it's only "sl…

Clearly all the overlapping points in the first graph are obscuring most of what's going on. This is a warning flag that the author may be trying color the facts to suit their argument. Not very clear what you are try to say: (1) What "standard deviation"? Perhaps you mean the standard error of the correlation coefficient is large relative to the its estimated value. Given the large number of data points it is likely…

"What "standard deviation"?

Typo: should have been "standard distribution"... Less colloquially: "a two-variable normal distribution."

"What does "ellipse isn't irregular" mean?"

Eccentricity. Sorry `fer the double negative.

"Kepler? The linked graph? How is this helpful in understanding what's going on?"

If you take the scattered data and draw an ellipse containing a fixed probability (presumably represented with some confidence, by the data shown), the ellipse would be more eccentric for the same enclosed probability, the higher the correlation. It's a single statistic and is analogous to Kepler's 2nd law ("a line joining a planet and the Sun sweeps out equal areas during equal intervals of time."), except with probability rather than area... (To contain the same probability, you have to adjust the ellipse's size, while holding its enclosed probability constant, and accounting for the new eccentricity, which isn't under our control. If the area is fixed and the distance changes, solve for speed... I think visually. Alternatively, think V=IR.; if V is constant and R is known, solve for I.). The link was just illustrative of any such scatter plot. It was the best I could find, but I agree, it could have been better.

I agree that the article is low on quantitative statistics though.

Re: The 50 million dollar lie

#80

Post by WalterBright »

colanderman wrote: post #59
WalterBright wrote: post #38

There is no objective way to measure teacher performance. Any evaluation method that can be written as a list of rules can and will quickly be gamed. The thing is, it's easy to find out who the best teachers are. Simply ask the students, parents, and staff. They all know who the good ones and the bad ones are - and that can't be gamed.

Your claim makes no sense. If the evaluation method measures what we want teachers to do, then it, by definition, can't be "gamed": if the measurement is high, then the teacher is doing what we want. Are you claiming that there's something special about teaching that makes teacher performance much more difficult, or even impossible, to quantify? If so, what is the point of teaching if one cannot measure results? Your…

You can't specify everything in the rules. For example, I know a case where management decided to hand out bonuses to the staff for minimizing inventory, because inventory costs money. The staff minimized inventory, all right, and got their bonuses. Meanwhile, the production line would regularly get halted because they'd run out of things like 5 cent resistors. It was a disaster.

The same company decided to rate programmers based on the bug list. A big chart was posted on the wall that showed the daily bug count. It wasn't a week before huge fights erupted over the bug count - over what was and was not a bug. The programmers quickly gamed it. They'd hide bugs, they'd refuse to fix one bug if that fix would produce some other minor bugs (e.g. if the bug was "feature X missing", then feature X is added, but had a couple issues with it, then X would not get fixed). They'd even add "bugs" blamed on other programmers and then "fix" them to get the credit.

Management gave up on that after two weeks and pulled the banner down.

Post reply on HN