Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The 50 million dollar lie

garyrubinstein.teachforus.org

Post reply on HN
Print view View original

91–100 of 159 posts

Re: The 50 million dollar lie

#91

Post by [deleted] »

colanderman wrote: post #59
WalterBright wrote: post #38

There is no objective way to measure teacher performance. Any evaluation method that can be written as a list of rules can and will quickly be gamed. The thing is, it's easy to find out who the best teachers are. Simply ask the students, parents, and staff. They all know who the good ones and the bad ones are - and that can't be gamed.

Your claim makes no sense. If the evaluation method measures what we want teachers to do, then it, by definition, can't be "gamed": if the measurement is high, then the teacher is doing what we want. Are you claiming that there's something special about teaching that makes teacher performance much more difficult, or even impossible, to quantify? If so, what is the point of teaching if one cannot measure results? Your…

[deleted]

Re: The 50 million dollar lie

#92

Post by yummyfajitas »

jcampbell1 wrote: post #85
kenjackson wrote: post #39

Earlier quoted context omitted.

Huh? The averaging doesn't give you only the positive correlation. It simply averages it. If the correlation was negative, you'd see that just as plainly. Averaging doesn't change the trend -- it just removes the volatility from the graph and makes the graph more readable (what's the density at y=x? I suspect it's a lot denser than he'd have you believe, but you can't tell from the chart at all). Sure, you could add…

You are missing the connection between these statistics and school HR policy. Sure there is a small positive correlation, but at what cost should we extract this benefit? Given this data, should we use value-add as a major component of teacher evaluations, and ultimately compensation/terminations? Giving employee evaluations that are 5% performance and 95% random chance is a recipe for an organization that no capable…

What form of evaluations do you believe perform better than VAM, and why?

Re: The 50 million dollar lie

#93

Post by yummyfajitas »

streptomycin wrote: post #45
kenjackson wrote: post #39

Earlier quoted context omitted.

Huh? The averaging doesn't give you only the positive correlation. It simply averages it. If the correlation was negative, you'd see that just as plainly. Averaging doesn't change the trend -- it just removes the volatility from the graph and makes the graph more readable (what's the density at y=x? I suspect it's a lot denser than he'd have you believe, but you can't tell from the chart at all). Sure, you could add…

The problem is that averaging can make a very small correlation look like a very strong correlation. You might be interested in this: http://en.wikipedia.org/wiki/Ecological_fallacy Nobody is arguing that a small correlation doesn't exist. They are arguing that a small correlation isn't very meaningful. Or are you saying that over 12 years of schooling you don't think you'd have any preference having your child taugh…

Or are you saying that you want to pick who to fire and who to promote based on a metric that we objectively know has only a very weak correlation with reality?

What alternatives do we have? Do you know of a more accurate metric?

Re: The 50 million dollar lie

#94

Post by DataWorker »

Using LOC to hire or fire programmers ignores important contextual factors, just as using test scores of students to hire or fire teachers ignores important contextual factors. What language is being used? How large is the project? Existing code? Platform? Bug count? Does the person contribute to the team in other ways?

In teacher evaluation you also have critically important contextual factors. What course is being taught? Are the students motivated? What's the classroom like? A lot of ELL kids in the class; does the teacher speak their language?

Many people see some amount of value in these imperfect metrics because they a) seek to measure the critical output (programmers are paid to make code and teachers are paid to make kids learn), and b) simple numeric metrics aren't biased by human fallibility. Popularity amongst peers won't have any impact on LOC or student test scores.

The author feels they shouldn't be used as 33% of the overall evaluation, which is the suggestion made by Gates.

In terms the specific points the author made about Gates' evidence in favor of their suggestion of 33%, I think he's basically right. I haven't read the original study and so I'm assuming for sake of discussion that he's conveying it correctly.

Over saturated scatterplots are not very useful, and in the hands of someone with no training or knowledge of basic statistics they may be dangerous, but they're not wrong. A trained consumer of research, seeing a plot like the author's first, will only be able to conclude that the correlation is not perfect. That rules out only a subset, and a subset that's fairly uncommon outside of the physical sciences, of the possible relationships between two variables. Not very useful, but not misleading unless there's an assumption that the number of data points is relatively small (which may be the case in some research contexts).

Averaging points and then taking the fake aggregated points and plotting those is wrong. It's using a tool to show variability in the wrong way. Averaging hides the variability that is supposed to be shown in a scatterplot. Why not average down to 2 points and draw the line then? Because that's wrong.

I'm thankful that one or two other people here understand, but a bit dismayed at the kneejerk groupthink.

Re: The 50 million dollar lie

#95

Post by crusso »

derpmaster wrote: post #81

This is the corporatist way. Everything is measured so it can be 'tweaked' by managers and technocrats because school is a business now. Having a bad teacher to me doesn't matter when any kid can turn on his phone or laptop and watch free MIT lectures, free Stanford lectures, free Khanacademy lectures, free Android and iOS developer courses all over youtube, ect. Gates obsession with teacher metrics would be better s…

Having a bad teacher to me doesn't matter when any kid can turn on his phone or laptop and watch free MIT lectures

Alternative self-study education is small compensation for having your child literally waste a school year in a teacher's class who has no clue how to keep control of the kids or desire to go over the material in class vs assigning everything for homework. In my daughter's case, her fourth grade teacher was too busy using that vaunted Internet access to check her facebook page.

Also, your "doesn't matter" argument would hold a lot more water if I didn't have to pay significant taxes to support such a lame public education system.

Re: The 50 million dollar lie

#96

Post by yummyfajitas »

danso wrote: post #10

I see the OP's point, that visual noise can be filtered in such a way as to create a more compelling trend, and perhaps the filtering that Gates has done has simplified the results...so, OK. Let's not look at charts, let's look at actual calculations that have the various factors accounted for...what are they? And I guess I'll throw in my two cents into the inevitable teachers-vs-data debate. I believe teachers may b…

Among the other problems is that administrators are just as prone to shortsightedness, and "the numbers" can be wielded by a petty/mediocre principal/district officer to apply pressure to a teacher.

The more we rely on objective numbers, the less a good teacher has to fear from a bad principal. The principal has no ability to make the teacher's students do worse than their statistical predictor, after all.

On the other hand, many other popular metrics (e.g. principal's opinion/coworker's opinion) can easily be tweaked by a principal who wants to put pressure on a teacher.

Re: The 50 million dollar lie

#97

Post by graeham »

lancewiggs wrote: post #74
graeham wrote: post #15

There are two problems with this article. First is that the "evidence" is from a scatter plot graph, with no statistical measurements attached to it. Stats really aren't something that you can rely on your eyes on. To me, it looks like a weak, positive correlation between previous year's score and current years score. The thing to look for is greater densities in the NE and SW quadrants than the NW and SE. Having a z…

Exactly. Charts are useful for eyeballing, but only the use of proper statistical tools is valid for showing trends. Charts can easily be very misleading. There seem to be a lot of comments about the charts on this page. Arguably basic statistics should be a part of every education - is that happening in the USA?

Can't comment on USA - in Canada we had an intro in high school then about a 1/2 course-credit module per year from second year until grad school in the engineering programs.

Re: The 50 million dollar lie

#98

Post by jacalata »

FourthProtocol wrote: post #5
kenjackson wrote: post #2

This is a surprising blog post in that I draw the complete opposite conclusion the author does. The author seems to think that the averaging hides volatility (which it does), which leads to incorrect conclusions drawn. Whereas to me it looks like it removes visual noise to show an actual trend. In his original scatter plot, because of the big ball in the middle it's easy to handwave and say, "look a random blob" -- b…

The other thing the author does is to derive absolute findings when the author himself admits to making an assumption about how the numbers were obtained. To me, using the word "lie" in the title is therefore nothing more than click-baiting. And on a separate note - there's a lot of anti-Gates sentiment flying around lately. Whether his methods are the best or not, he's using his own money . I find it really difficul…

People are arguing that he is using his own money to actively damage society. I'm not sure if I agree with that, but to say you can't criticize someone for what they do with their money is ridiculous - as an extreme, he could use his money to poison people, or to lobby for life sentences for marijuana use.

Re: The 50 million dollar lie

#99

Post by yummyfajitas »

WalterBright wrote: post #80
colanderman wrote: post #59

Earlier quoted context omitted.

Your claim makes no sense. If the evaluation method measures what we want teachers to do, then it, by definition, can't be "gamed": if the measurement is high, then the teacher is doing what we want. Are you claiming that there's something special about teaching that makes teacher performance much more difficult, or even impossible, to quantify? If so, what is the point of teaching if one cannot measure results? Your…

You can't specify everything in the rules. For example, I know a case where management decided to hand out bonuses to the staff for minimizing inventory, because inventory costs money. The staff minimized inventory, all right, and got their bonuses. Meanwhile, the production line would regularly get halted because they'd run out of things like 5 cent resistors. It was a disaster. The same company decided to rate prog…

For teachers, the metric is (roughly speaking) "% of students capable of multiplying/dividing numbers up to 4 digits in a standardized test setting". How do you game this metric?

The fact that one company used a couple of bad objective metrics doesn't mean all objective metrics are bad. They are used with a great deal of success in many fields. Sales people are paid on commission, traders are paid proportionally to (risk adjusted) profits, etc. All it means is that if you set the wrong goals for your company, you'll probably succeed at the wrong thing.

So tell us - is "maximize the % of students capable of multiplying/dividing 4 digit numbers" the wrong goal? If so, what is the right goal, and why can't it be measured?

Re: The 50 million dollar lie

#100

Post by spikels »

streptomycin wrote: post #63
ef4 wrote: post #61

Earlier quoted context omitted.

The blog post doesn't have any statistics to ignore. Graphs are not statistics. Why doesn't he give us a correlation coefficient on his big blue blob graph?

You criticize a short blog post for something that is not done in the Gates publication. Do you have some bias, perhaps? Why does Gates not even give us a scatter plot, let alone a correlation coefficient? This is the critical point raised by the blog.

There are quite a few statistics reported in the Gates Foundation publication. Take a look for yourself:

http://www.metproject.org/downloads/MET_Validating_Using_Ran...

Post reply on HN