Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The 50 million dollar lie

garyrubinstein.teachforus.org

Post reply on HN
Print view View original

101–110 of 159 posts

Re: The 50 million dollar lie

#101

Post by DataWorker »

colanderman wrote: post #59
WalterBright wrote: post #38

There is no objective way to measure teacher performance. Any evaluation method that can be written as a list of rules can and will quickly be gamed. The thing is, it's easy to find out who the best teachers are. Simply ask the students, parents, and staff. They all know who the good ones and the bad ones are - and that can't be gamed.

Your claim makes no sense. If the evaluation method measures what we want teachers to do, then it, by definition, can't be "gamed": if the measurement is high, then the teacher is doing what we want. Are you claiming that there's something special about teaching that makes teacher performance much more difficult, or even impossible, to quantify? If so, what is the point of teaching if one cannot measure results? Your…

Which is why we have objective measures for programmers.

Re: The 50 million dollar lie

#102

Post by DanBC »

yummyfajitas wrote: post #99
WalterBright wrote: post #80

Earlier quoted context omitted.

You can't specify everything in the rules. For example, I know a case where management decided to hand out bonuses to the staff for minimizing inventory, because inventory costs money. The staff minimized inventory, all right, and got their bonuses. Meanwhile, the production line would regularly get halted because they'd run out of things like 5 cent resistors. It was a disaster. The same company decided to rate prog…

For teachers, the metric is (roughly speaking) "% of students capable of multiplying/dividing numbers up to 4 digits in a standardized test setting". How do you game this metric? The fact that one company used a couple of bad objective metrics doesn't mean all objective metrics are bad. They are used with a great deal of success in many fields. Sales people are paid on commission, traders are paid proportionally to (…

> "% of students capable of multiplying/dividing numbers up to 4 digits in a standardized test setting". How do you game this metric?

You use the time that you used to use for teaching reading and devote it to teaching methods for multiplying and dividing numbers up to 4 digits in a standardized test setting.

You might say this isn't gaming the test, it's just a stupid way for schools to optimise their results for that metric. But that metric is mechanical; there's no measurement for deep understanding of the principles, just success or failure for applying mechanical rules by rote.

Most US standardised test settings use machine marked multiple choices. I'm told that there are lessons for 'bubbling in' - lesson on how to fill in the multiple choice answer bubbles to ensure fewer mis-marked answers.

Re: The 50 million dollar lie

#103

Post by pdonis »

kenjackson wrote: post #39
pdonis wrote: post #4

Earlier quoted context omitted.

You're missing the point. The author agreed that there is a slight positive correlation in the data. But the averaging makes it look like there is a strong positive correlation in the data, which there isn't. In other words: without the averaging, you are seeing the full implications of the data: slight positive correlation, but a lot of scatter--i.e., not much meat there. With the averaging, you are seeing only the…

Huh? The averaging doesn't give you only the positive correlation. It simply averages it. If the correlation was negative, you'd see that just as plainly. Averaging doesn't change the trend -- it just removes the volatility from the graph and makes the graph more readable (what's the density at y=x? I suspect it's a lot denser than he'd have you believe, but you can't tell from the chart at all). Sure, you could add…

you'd rather have your child by a teacher with a high teacher score than a low one

Not necessarily. The very weak correlation means that the difference in this score only accounts for a very small amount of the difference in outcome; and even that claim assumes that the observed correlation is evidence of a causal link, which is not necessarily the case. You can't usefully conclude anything from this data except that more investigation is needed.

Re: The 50 million dollar lie

#104

Post by wisty »

streptomycin wrote: post #45
kenjackson wrote: post #39

Earlier quoted context omitted.

Huh? The averaging doesn't give you only the positive correlation. It simply averages it. If the correlation was negative, you'd see that just as plainly. Averaging doesn't change the trend -- it just removes the volatility from the graph and makes the graph more readable (what's the density at y=x? I suspect it's a lot denser than he'd have you believe, but you can't tell from the chart at all). Sure, you could add…

The problem is that averaging can make a very small correlation look like a very strong correlation. You might be interested in this: http://en.wikipedia.org/wiki/Ecological_fallacy Nobody is arguing that a small correlation doesn't exist. They are arguing that a small correlation isn't very meaningful. Or are you saying that over 12 years of schooling you don't think you'd have any preference having your child taugh…

The point is, you shouldn't worry too much about teacher performance.

If it's only loosely correlated, measuring value-added then punishing / rewarding teachers based it won't be very accurate. It's like lines of code - programmer talent is probably weakly correlated with lots of lines of code, but you don't want to reward or punish them based on LoC because it will lead to pathological behavior.

You should look at other things to alter (which are less easy to game once you start trying to control them), like class size, course materials, assessment style, how the teacher actually teaches, etc.

Re: The 50 million dollar lie

#105

Post by streptomycin »

yummyfajitas wrote: post #93
streptomycin wrote: post #45

Earlier quoted context omitted.

The problem is that averaging can make a very small correlation look like a very strong correlation. You might be interested in this: http://en.wikipedia.org/wiki/Ecological_fallacy Nobody is arguing that a small correlation doesn't exist. They are arguing that a small correlation isn't very meaningful. Or are you saying that over 12 years of schooling you don't think you'd have any preference having your child taugh…

Or are you saying that you want to pick who to fire and who to promote based on a metric that we objectively know has only a very weak correlation with reality? What alternatives do we have? Do you know of a more accurate metric?

One alternative is to not use these flawed/complicated/expensive metrics. Imagine if I came up with a metric that had an 0.1 correlation coefficient with your performance at your job. If you're a programmer, maybe that'd be something like measuring the lines of code written per day. Would you put much faith in that? Of course not, there would be myriad problems with such a system. Same thing here.

The status quo in teaching is to evaluate teachers based on qualitative observations from management, same as it is in most other industries. As the same blogger points out in another post http://garyrubinstein.teachforus.org/2013/01/13/50-million-3... "Conspicuously missing from the various weighting schemes they compare is one with 100% classroom observations. As this is what many districts currently do and since this report is supposed to guide those who are designing new systems, wouldn’t it be scientifically necessary to include the existing system as the ‘control’ group? As implementing a change is a costly and difficult process, shouldn’t we know what we could expect to gain over the already existing system?"

The status quo also has the advantage of not being ridiculously expensive and complicated. Ask Bill Gates to release the data if you're curious about its performance, but given what I've seen, I'd be shocked if it was terribly different than these complicated new metrics.

Re: The 50 million dollar lie

#106

Post by btilly »

ColinWright wrote: post #9

I suspect there'll be a lot of differing opinions on this, and I'm looking forward to seeing the discussion. What would be helpful would be if people could say how much experience they have in hard statistics, and how much what they say is driven a priori from the data. I know that hackers, in particular, have real problems with "Argument from Authority", but stats is one place where it's really, really easy to go wr…

Statistics, like engineering, is done with numbers. When you attempt to do it without numbers, that's called "opinion".

I have not tried to look at the numbers. Here is my opinion.

My experience of school was that a small minority of teachers are truly excellent, and a small minority were horrible. The truly excellent ones are not distinguished so much by what happened in their class as by what happened in the following classes, and what classes they left people excited about taking. The terrible ones, by contrast, showed up as poor performance all around.

The described data set has a clear correlation. Whether or not the correlation is meaningful in individual cases is much less clear. My initial approach would be to try to fit a hierarchical Bayesian model to a data set, then use the resulting model to come back with predictions about individual teachers. Teachers whom, after several measurements, are overwhelmingly identified as terrible should be removed. If you find a population of teachers who show up as superstars, they should be subjected to further study to see if we can predict the quality of incoming teachers, and to see if we can learn lessons from them that improve other teachers.

However this is a well-studied problem. I'm sure someone has tried something like this. I am sure that there are a lot of vested interests. I have not attempted to evaluate work in this area, and I have no opinion on how good it is.

Re: The 50 million dollar lie

#107

Post by yummyfajitas »

DanBC wrote: post #102
yummyfajitas wrote: post #99

Earlier quoted context omitted.

For teachers, the metric is (roughly speaking) "% of students capable of multiplying/dividing numbers up to 4 digits in a standardized test setting". How do you game this metric? The fact that one company used a couple of bad objective metrics doesn't mean all objective metrics are bad. They are used with a great deal of success in many fields. Sales people are paid on commission, traders are paid proportionally to (…

> "% of students capable of multiplying/dividing numbers up to 4 digits in a standardized test setting". How do you game this metric? You use the time that you used to use for teaching reading and devote it to teaching methods for multiplying and dividing numbers up to 4 digits in a standardized test setting. You might say this isn't gaming the test, it's just a stupid way for schools to optimise their results for th…

You use the time that you used to use for teaching reading and devote it to teaching methods for multiplying and dividing numbers up to 4 digits in a standardized test setting.

Clever - you caught me. I got lazy and didn't feel like typing up all the goals of a 3'rd grade education system into an HN comment, preferring only to provide a simple example and hoping that a reasonable reader would extrapolate.

Mea culpa - I'll stop assuming reasonable readers in the future.

Or are you actually suggesting that the school system might forget to include reading when defining their goals?

Re: The 50 million dollar lie

#108

Post by patio11 »

yummyfajitas wrote: post #72
feral wrote: post #7

I know nothing about the domain of teacher measurement, and have no opinion on it. But the first chart in the blog post, with the big mass of blue on it, is surely not the strong evidence of weak correlation, that the author is making it out to be? There could still be a strong correlation in that data, even if it looks like a blob of blue - because there are so many data points on that chart, that we can no longer t…

I'm really surprised to see Gary Rubinstein continuing to post bad graphs which obscure the data. 8 months ago I wrote a blog post about why you should avoid scatterplots, and used his graphs as examples of what not to do. http://www.chrisstucchio.com/blog/2012/dont_use_scatterplots... He is certainly aware of my post - he even commented on it and I offered him some constructive suggestions (such as adding alpha if h…

That is common for pre-moderated comments on blogs rather than evidence of hell banning, fwiw.

Re: The 50 million dollar lie

#109

Post by streptomycin »

spikels wrote: post #100
streptomycin wrote: post #63

Earlier quoted context omitted.

You criticize a short blog post for something that is not done in the Gates publication. Do you have some bias, perhaps? Why does Gates not even give us a scatter plot, let alone a correlation coefficient? This is the critical point raised by the blog.

There are quite a few statistics reported in the Gates Foundation publication. Take a look for yourself: http://www.metproject.org/downloads/MET_Validating_Using_Ran...

I never said they don't report "quite a few statistics". I said they don't report this one specific particularly important statistic, and instead obscure it behind a ridiculous averaging procedure. This was either done because they are incompetent or because they are purposely trying to mislead. That's the whole point of TFA.

Re: The 50 million dollar lie

#110

Post by crusso »

IEatShortPeople wrote: post #26

Even if Bill Gates isn't doing the best job, every time I see one of these articles, it appears that there isn't any advice on better ways to grade teachers. Did I miss that in the article? Are there a lot of other people in the field that are doing better work? (I would really like to read if anyone can provide links to papers to other ways of analyzing teachers)

Did I miss that in the article

The blog link to this thread is a propaganda site. It conspicuously doesn't reference any of the original materials that it is attempting (weakly) to refute.

There was an article on HN last week that referenced a more source material article about the project.

Here's the link to the METProject's web site: http://www.metproject.org/

Post reply on HN