Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The 50 million dollar lie

garyrubinstein.teachforus.org

Post reply on HN
Print view View original

151–159 of 159 posts

Re: The 50 million dollar lie

#151

Post by yummyfajitas »

streptomycin wrote: post #150
yummyfajitas wrote: post #148

Earlier quoted context omitted.

The 0.3 correlation described in the article is the correlation between a teacher's VAM score in a single class last year and this year. The author is arguing that because the measurement is noisy, we should ignore it. This is silly - it just means a single class-year's point estimate is noisy, and multiple class-years must be combined to form an accurate estimate. If you had 1/3 of your salary determined by the LOC…

The author is arguing that because the measurement is noisy, we should ignore it. Not quite.. Indeed - if 1/3 of my salary was determined by the number of kids in my class who can read better than when they are predicted to read at this age, I'd definitely try to make sure that reading skills improved. Keep in mind that "score on a reading test" and "reading skills" are not the same things. For instance, one would be…

Keep in mind that "score on a reading test" and "reading skills" are not the same things.

True. Score on a reading test might be lower than actual reading skills due to a lack of test taking skills.

Then again, I can't think of any measurable quantity more highly correlated with reading skills than scoring high on a reading test. Do you have evidence that "principal's opinion of teacher" is better?

...we have a metric that very loosely correlates...

You are confused. GR shows that year on year correlations are weak (0.3), not that correlation between reading test and reading skills are weak.

You are correct that a certain nonzero amount of teaching test skills will be effective at improving scores. I see no reason to believe this amount will be large - do you?

Again, standardized tests are not the only gameable metric. Principal/student opinion is as well. Why do you believe they are less gameable and more accurate than directly measuring student ability?

Re: The 50 million dollar lie

#152

Post by streptomycin »

yummyfajitas wrote: post #151
streptomycin wrote: post #150

Earlier quoted context omitted.

The author is arguing that because the measurement is noisy, we should ignore it. Not quite.. Indeed - if 1/3 of my salary was determined by the number of kids in my class who can read better than when they are predicted to read at this age, I'd definitely try to make sure that reading skills improved. Keep in mind that "score on a reading test" and "reading skills" are not the same things. For instance, one would be…

Keep in mind that "score on a reading test" and "reading skills" are not the same things. True. Score on a reading test might be lower than actual reading skills due to a lack of test taking skills. Then again, I can't think of any measurable quantity more highly correlated with reading skills than scoring high on a reading test. Do you have evidence that "principal's opinion of teacher" is better? ...we have a metri…

It's not that I think this is the only flawed metric or that there are other methods that are quantifiably better. It's that we've spent ridiculous sums of time and money on this complicated new metric, despite some pretty clear arguments against it, and we still are very far away from actually proving that it is better than the status quo, even if that status quo obviously falls somewhat short of perfection.

It's like.. how much money and effort do you think is worthwhile to spend on something with such uncertain prospects? There are tons of great grant proposals to funding agencies which are rejected. Maybe we would be better off funding some of those. Opportunity costs exist, and it seems like undue sums of money are being wasted on this stuff.

Re: The 50 million dollar lie

#153

Post by streptomycin »

yummyfajitas wrote: post #147
streptomycin wrote: post #125

Earlier quoted context omitted.

I do find it odd that teachers would rather be judged based on one or two people observing their classroom and ignoring actual student output, rather than raw numbers. As a developer I'd much rather be evaluated on some metric I could optimize for (I think some metric measuring feature value/bugs/fix rate/etc...), rather than just my manager watching me code/debug a couple of times per month. Okay, I really don't und…

You'd really prefer it is, say, 1/3 of your salary was determined by the number of lines of code you wrote? I was a quant trader for a while. A considerable chunk of my salary was determined by profit, hardly unreasonable. Measuring developers in other areas is difficult because of heterogeneous goals - last year I built a search engine, this year I'm statistically tracking user behavior. Hard to compare one to the o…

I don't think it's obvious that these teaching metrics correlate better with teaching ability than LOC would correlate with programming ability. I'd expect weak correlations in both cases. I'm just speculating at this point, though.

Re: The 50 million dollar lie

#154

Post by spikels »

keithpeter wrote: post #143
spikels wrote: post #140

Earlier quoted context omitted.

You are correct that if students were assigned in some biased manner it could invalidate any relationship between teacher quality and subsequent student performance. That is exactly what makes this study interesting. All the students were assigned randomly! Much like in a randomized medical study this allows the establishment of a causal relationship.

" All the students were assigned randomly! Much like in a randomized medical study this allows the establishment of a causal relationship. " Two thoughts... 1) Ethics: You put some children with dyslexia or with challenging behaviour with newly trained teachers with little experience? Did their parents have any say in this? Was there an ethics process? Was there support available for newly trained teachers meeting ma…

1) Same students, same teachers and same support in same schools but the assignment of students and teachers to classes was randomized. Consent was obtained from principals, teachers and parents.

2) Randomized experiments can establish causation if done properly. This is how medical treatments are evaluated. The theory is simple: teachers that were more effective in the past will be more effective in the future and using objective metrics (tests, classroom observation and surveys) we can measure teacher effectiveness. This certainly positivist but more important likely repeatable.

You seem like you are interested in this but I get the sense you already have strong views. Maybe you should put those aside for a moment and take a closer look at this new and interesting study you might learn something:

http://www.metproject.org/reports.php

Re: The 50 million dollar lie

#155

Post by randomknowledge »

yummyfajitas wrote: post #146
randomknowledge wrote: post #142

Earlier quoted context omitted.

Okay hows this: unless the author of the original blog post is deliberately deceiving the audience but putting a bunch of points on top of each other then one can tell there is no line which most of the points lie near. I agree that a scatter plot is not the best for showing data with that many data points, but frankly it's kind of irrelevant. The point the author was making was just that it isn't that hard to cook u…

It's not a matter of being deliberate. He's just being lazy, using the standard excel scatterplot. To fix that graph, he needs to carefully choose an opacity, which he didn't do. See the discussion here: http://news.ycombinator.com/item?id=4027337 I stand by the claim made in my blog post. Don't use scatterplots, use a density plot instead. Incidentally, according to a comment the author made, the correlation is actu…

I agree he could have presented his data better. I also recognize that there is some correlation. None the less, the story was about how averaging can be used to make a correlation appear strong than it is, and going from 0.3 to 0.99 certainly meets that criteria. I was responding to the claim that the data might already have a strong correlation (presumably on the order of 0.99), and I was arguing that no "natural" strongly correlated data would have a scatter plot like that.

I might add that discussions like these, Illustrate a broader problem with the hacker news community. Rather than discussing the facts of the story, many of the posts are instead about the less relevant detail about how the author chose to present them, despite the fact the the authors point is still effectively made (and if the 0.3 were included there wouldn't have been any doubt)

BTW: tmoertel did some analysis on the data: http://news.ycombinator.com/item?id=5125851 and the correlation appears to be around 0.2, which is pretty weak.

Re: The 50 million dollar lie

#156

Post by nialo »

yummyfajitas wrote: post #124
DanBC wrote: post #119

Earlier quoted context omitted.

> up all the goals of a 3'rd grade education system Can all the goals of a 3rd grade education system be reduced to a purely mechanical list of stuff? Let's try applying your example of arithmetic to reading. What are the goals? To get children to read individual words? Or to get children to read a sentence, and obtain meaning from it? If it's just to read words, do you get those words from a defined vocab? Should th…

Can all the goals of a 3rd grade education system be reduced to a purely mechanical list of stuff? Yes, I would hope that a multi-million dollar enterprise can clearly define their goals. What are the goals? To get children to read individual words? Or to get children to read a sentence, and obtain meaning from it? I don't know off the top of my head whether the latter should be learned by 3rd grade. Ultimately setti…

Regarding stating goals clearly: The problem is that the real goal is something like "maximize future student happiness" or perhaps "maximize future student income", and any set of test goals is therefore necessarily an approximation. I think that if you want to make this argument, and it seems like a reasonable one, the thing that actually needs to be shown is that some specific set of easy to write down test scores does a reasonable job as an approximation, or at least could in principle if one could just find the correct test.

Re: The 50 million dollar lie

#157

Post by yummyfajitas »

nialo wrote: post #156
yummyfajitas wrote: post #124

Earlier quoted context omitted.

Can all the goals of a 3rd grade education system be reduced to a purely mechanical list of stuff? Yes, I would hope that a multi-million dollar enterprise can clearly define their goals. What are the goals? To get children to read individual words? Or to get children to read a sentence, and obtain meaning from it? I don't know off the top of my head whether the latter should be learned by 3rd grade. Ultimately setti…

Regarding stating goals clearly: The problem is that the real goal is something like "maximize future student happiness" or perhaps "maximize future student income", and any set of test goals is therefore necessarily an approximation. I think that if you want to make this argument, and it seems like a reasonable one, the thing that actually needs to be shown is that some specific set of easy to write down test scores…

...the thing that actually needs to be shown is that some specific set of easy to write down test scores does a reasonable job as an approximation...

All we really need is to believe that it's a better approximation than the alternative, which is currently something along the lines of 0.5 x Principal's Opinion + 0.5 x Union Seniority (at best).

Re: The 50 million dollar lie

#158

Post by nialo »

yummyfajitas wrote: post #157
nialo wrote: post #156

Earlier quoted context omitted.

Regarding stating goals clearly: The problem is that the real goal is something like "maximize future student happiness" or perhaps "maximize future student income", and any set of test goals is therefore necessarily an approximation. I think that if you want to make this argument, and it seems like a reasonable one, the thing that actually needs to be shown is that some specific set of easy to write down test scores…

...the thing that actually needs to be shown is that some specific set of easy to write down test scores does a reasonable job as an approximation... All we really need is to believe that it's a better approximation than the alternative, which is currently something along the lines of 0.5 x Principal's Opinion + 0.5 x Union Seniority (at best).

I personally think I agree that it is a better approximation. I'm just trying to help get the discussion pointed in the right direction.

My main point is that I think it's better to frame this as "this is a better approximation" than as "reading is the goal". That's all.

Re: The 50 million dollar lie

#159

Post by keithpeter »

spikels wrote: post #154
keithpeter wrote: post #143

Earlier quoted context omitted.

" All the students were assigned randomly! Much like in a randomized medical study this allows the establishment of a causal relationship. " Two thoughts... 1) Ethics: You put some children with dyslexia or with challenging behaviour with newly trained teachers with little experience? Did their parents have any say in this? Was there an ethics process? Was there support available for newly trained teachers meeting ma…

1) Same students, same teachers and same support in same schools but the assignment of students and teachers to classes was randomized. Consent was obtained from principals, teachers and parents. 2) Randomized experiments can establish causation if done properly. This is how medical treatments are evaluated. The theory is simple: teachers that were more effective in the past will be more effective in the future and u…

"You seem like you are interested in this but I get the sense you already have strong views."

I will read the reference, however, to those of us in Europe, please remember that this methodology resembles a group of earnest steam punks analysing flies in amber with brass and ebony microscopes on an ice floe that is breaking up... I'm suggesting that we suspect positivist descriptions do not capture the important values in education. 90% of what children learn is learned outside school I suggest.

I am prepared to be astounded with contrary evidence.

Post reply on HN