Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The 50 million dollar lie

garyrubinstein.teachforus.org

Post reply on HN
Print view View original

41–50 of 159 posts

Re: The 50 million dollar lie

#41

Post by Alex3917 »

jellicle wrote: post #30
Alex3917 wrote: post #8

Earlier quoted context omitted.

"Whereas to me it looks like it removes visual noise to show an actual trend." Except for that in this case the noise is more important than the trend. Think about it, if you're firing or sanctioning perhaps 30%+ of teachers each year for no reason, then only complete morons would go into teaching. It's the same as airport security, where a .1% false positive rate is unacceptable, whereas a 75% false negative rate is…

Currently the number of public school teachers fired or sanctioned per year is much less than 0.1%. Which is a bit different than 30%. So I'm thinking they're not too worried about this.

"Currently the number of public school teachers fired or sanctioned per year is much less than 0.1%."

This is comparing apples and oranges.

Re: The 50 million dollar lie

#42

Post by jellicle »

itg wrote: post #37
jellicle wrote: post #31

So if the teacher evaluation method is alleged to be inferior, please provide a new evaluation method which is superior to it. Can't? Won't? Then be quiet.

The idea would be not to waste numerous resources and money implementing a broken idea and spend some time coming up with a better one. Why put the education of our children and the careers of good teachers on the line with such a poorly thought out and half-assed idea?

If you think that no time has been spent coming up with this idea, you are coming very late to the party. This debate is decades old, at a minimum. Dates from 1971, says Wikipedia.

http://en.wikipedia.org/wiki/Value-added_modeling

It's not something that "Bill Gates" or anyone else made up yesterday. It's a state-of-the-art approach to a difficult problem. If you have a better way, let's hear it. Otherwise, be quiet.

Re: The 50 million dollar lie

#43

Post by thedufer »

I'm confused as to why the "‘raw’ score (for value added, this is a number between -1 and +1)" goes up to ~1.6 in 2009-2010 and up to ~1.1 in 2008-2009. Why does the data go outside the ranges he states? I find it hard to believe someone who can't read something so simple off his own graph.

Re: The 50 million dollar lie

#44

Post by absherwin »

ColinWright wrote: post #9

I suspect there'll be a lot of differing opinions on this, and I'm looking forward to seeing the discussion. What would be helpful would be if people could say how much experience they have in hard statistics, and how much what they say is driven a priori from the data. I know that hackers, in particular, have real problems with "Argument from Authority", but stats is one place where it's really, really easy to go wr…

I'll bite. I won't discuss my experience in my posts since I don't want to argue from authority but will do so below. But first, I should point out that you're committing a similar sin to that which is alleged in the article. What you really care about is the correctness of an argument. You hypothesize that formal training is an important indicator of correctness. Presumably there's also noise in that scatterplot but we don't even have it. Since it's the best thing available, you are choosing to rely on it.

As for my formal stats training: None beyond High School. That said, I've worked with and interviewed many with far more training. One thing, I learned is that most people don't have great intuition for statistical problems and it's not terribly highly correlated with years of schooling. I've met PhD's in economics and statistics who have made basic conceptual errors as well as those with less training who were more reliable.

So while training may be correlated with accuracy, you should still demand a well-reasoned argument and think critically about it.

Re: The 50 million dollar lie

#45

Post by streptomycin »

kenjackson wrote: post #39
pdonis wrote: post #4

Earlier quoted context omitted.

You're missing the point. The author agreed that there is a slight positive correlation in the data. But the averaging makes it look like there is a strong positive correlation in the data, which there isn't. In other words: without the averaging, you are seeing the full implications of the data: slight positive correlation, but a lot of scatter--i.e., not much meat there. With the averaging, you are seeing only the…

Huh? The averaging doesn't give you only the positive correlation. It simply averages it. If the correlation was negative, you'd see that just as plainly. Averaging doesn't change the trend -- it just removes the volatility from the graph and makes the graph more readable (what's the density at y=x? I suspect it's a lot denser than he'd have you believe, but you can't tell from the chart at all). Sure, you could add…

The problem is that averaging can make a very small correlation look like a very strong correlation. You might be interested in this: http://en.wikipedia.org/wiki/Ecological_fallacy

Nobody is arguing that a small correlation doesn't exist. They are arguing that a small correlation isn't very meaningful.

Or are you saying that over 12 years of schooling you don't think you'd have any preference having your child taught by teachers with a score in the lowest 5%ile versus a teacher with a predicted score in the top 5%ile?

If you base those percentiles on a metric that has a very low correlation with performance, then there won't be much of a difference in outcome.

Or are you saying that you want to pick who to fire and who to promote based on a metric that we objectively know has only a very weak correlation with reality?

Re: The 50 million dollar lie

#46

Post by keithpeter »

"In New York City, the value-added score is actually not based on comparing the scores of a group of students from one year to the next, but on comparing the ‘predicted’ scores of a group of students to what those students actually get. The formula to generate this prediction is quite complicated, but the main piece of data it uses is the actual scores that the group of students got in the previous year."

http://garyrubinstein.teachforus.org/2012/03/06/analyzing-re...

Some qualitative comments (from the UK)

An experienced and resourceful teacher may be allocated more 'problem' students than a relatively new teacher as a matter of policy. Teaching students at the extremes of the ability range is harder than the median as the reasons for (say) severe under achievement can be very diverse (learning difficulties, home situation, undiagnosed medical issues - diabetes in my recent experience with one student - and so on) and each student may need a different approach.

I have taught in a few settings over the years. The best teaching happens in my experience in settings where people work together. Performance related pay based on dodgy statistics in an atmosphere of public ridicule does not seem the best way to encourage sharing of practice and teamwork!

"Good luck with that" as I believe the late Mr Jobbs used to say to competitors.

Re: The 50 million dollar lie

#47

Post by tmoertel »

ColinWright wrote: post #33
tmoertel wrote: post #27

Earlier quoted context omitted.

Do you have a convenient link to the data set you used to generate the first plot in your article?

I think you're making a mistake that I see over and over again here on HN. Not my data, not my article. I just submitted it here.

Sorry about the mix-up.

Anyway, here's the data for anyone interested: http://www.ny1.com/content/top_stories/156599/now-available-...

Re: The 50 million dollar lie

#48

Post by streptomycin »

jellicle wrote: post #42
itg wrote: post #37

Earlier quoted context omitted.

The idea would be not to waste numerous resources and money implementing a broken idea and spend some time coming up with a better one. Why put the education of our children and the careers of good teachers on the line with such a poorly thought out and half-assed idea?

If you think that no time has been spent coming up with this idea, you are coming very late to the party. This debate is decades old, at a minimum. Dates from 1971, says Wikipedia. http://en.wikipedia.org/wiki/Value-added_modeling It's not something that "Bill Gates" or anyone else made up yesterday. It's a state-of-the-art approach to a difficult problem. If you have a better way, let's hear it. Otherwise, be quiet.

You: "Yeah, okay, this method doesn't work well enough to accomplish anything and it's ridiculously expensive. HOWEVER, it's been used for decades, and I don't know any other methods that actually work. So let's continue pissing money down the drain."

Re: The 50 million dollar lie

#49

Post by streptomycin »

absherwin wrote: post #34

To build intuition, let's consider a hypothetical society. We divide a group of people in two. The first group rolls a die with 100 sides labeled from 1 to 100 in even increments. The second group rolls a die with labeled 1 to 110 in even increments. The roll determines their annual ability level which is unknowable. If you were hiring, which group would you prefer? Is it fair to the second group? Note that this isn'…

To provide a contrary hypothetical...

Imagine if one group had 55% of its dice labeled up to 110 and 45% up to 100, and the other group had the opposite. So if all you use for hiring is the group membership, you're going to very arbitrarily miss out on a lot of 110s and hire a lot of 100s. And now also imagine that this process of identifying who is in what group is ridiculously expensive.

Re: The 50 million dollar lie

#50

Post by colanderman »

feral wrote: post #7

I know nothing about the domain of teacher measurement, and have no opinion on it. But the first chart in the blog post, with the big mass of blue on it, is surely not the strong evidence of weak correlation, that the author is making it out to be? There could still be a strong correlation in that data, even if it looks like a blob of blue - because there are so many data points on that chart, that we can no longer t…

Not to mention that he says "this is a number between -1 and +1" and the graph clearly has data points outside that range.
Post reply on HN