Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The 50 million dollar lie

garyrubinstein.teachforus.org

Post reply on HN
Print view View original

111–120 of 159 posts

Re: The 50 million dollar lie

#111

Post by DataWorker »

yummyfajitas wrote: post #99
WalterBright wrote: post #80

Earlier quoted context omitted.

You can't specify everything in the rules. For example, I know a case where management decided to hand out bonuses to the staff for minimizing inventory, because inventory costs money. The staff minimized inventory, all right, and got their bonuses. Meanwhile, the production line would regularly get halted because they'd run out of things like 5 cent resistors. It was a disaster. The same company decided to rate prog…

For teachers, the metric is (roughly speaking) "% of students capable of multiplying/dividing numbers up to 4 digits in a standardized test setting". How do you game this metric? The fact that one company used a couple of bad objective metrics doesn't mean all objective metrics are bad. They are used with a great deal of success in many fields. Sales people are paid on commission, traders are paid proportionally to (…

A few come to mind:

cheating (see Atlanta Journal-Constitution series on this)

encouraging absenteeism amongst low performers

damaging the test materials of low performing students to get their scores excluded

getting low performers into IEPs so they don't test, take the alternate or receive accommodations

sabotage other teachers in various ways

vie for the most favorable students by pulling strings with the administration

I'm sure if there's actual motivation one might find even more.

Re: The 50 million dollar lie

#112

Post by yummyfajitas »

streptomycin wrote: post #105
yummyfajitas wrote: post #93

Earlier quoted context omitted.

Or are you saying that you want to pick who to fire and who to promote based on a metric that we objectively know has only a very weak correlation with reality? What alternatives do we have? Do you know of a more accurate metric?

One alternative is to not use these flawed/complicated/expensive metrics. Imagine if I came up with a metric that had an 0.1 correlation coefficient with your performance at your job. If you're a programmer, maybe that'd be something like measuring the lines of code written per day. Would you put much faith in that? Of course not, there would be myriad problems with such a system. Same thing here. The status quo in t…

One alternative is to not use these flawed/complicated/expensive metrics.

So replace a metric with 0.1 correlation with one having 0.0 correlation?

If you are advocating that we should use the school principal's opinion rather than VAM, why do you believe opinion is superior? Do you have evidence that principal's opinion has a higher correlation with student outcomes than VAM?

Re: The 50 million dollar lie

#113

Post by FourthProtocol »

jacalata wrote: post #98
FourthProtocol wrote: post #5

Earlier quoted context omitted.

The other thing the author does is to derive absolute findings when the author himself admits to making an assumption about how the numbers were obtained. To me, using the word "lie" in the title is therefore nothing more than click-baiting. And on a separate note - there's a lot of anti-Gates sentiment flying around lately. Whether his methods are the best or not, he's using his own money . I find it really difficul…

People are arguing that he is using his own money to actively damage society. I'm not sure if I agree with that, but to say you can't criticize someone for what they do with their money is ridiculous - as an extreme, he could use his money to poison people, or to lobby for life sentences for marijuana use.

I said Bill Gates, and not someone. And while I don't know how people think he's damaging society, if the article being discussed here is representative then I remain unconvinced. This is not an invitation to enlighten me. I'm not wasting my time reading yet another opinion, assumption or half-baked arm-chair allegation. The investment dollars and opinion of the likes of Warren Buffett are a much more compelling endorsement to me than (I mean this respectfully) members of HN.

Re: The 50 million dollar lie

#114

Post by spikels »

vy8vWJlco wrote: post #88
spikels wrote: post #86

Earlier quoted context omitted.

Sorry don't mean to belabor this point but I think "standard distribution" (changed from standard deviation) is still incorrect. "Standard" referers to standardized parameters (i.e. mean, variance) as in "standard normal" which has a mean of zero and variance of one (and standard deviation of one). Certainly not the case here. Maybe you mean "bivariate normal with positive correlation"? http://en.wikipedia.org/wiki/M…

Yes, you are right. I wasn't being especially rigorous, and technically correct is the "best kind" of correct. :)

No worries. Statistics can be very confusing. Part of the reason it is so confusing is all the sloppy usage we come across in the media and on the internet.

Everyone wants to use statistics as a tool to bolster their point of view but don't want to bother to actually learn enough statistics to deal with the situation they are trying to analyze.

Re: The 50 million dollar lie

#115

Post by yummyfajitas »

DataWorker wrote: post #111
yummyfajitas wrote: post #99

Earlier quoted context omitted.

For teachers, the metric is (roughly speaking) "% of students capable of multiplying/dividing numbers up to 4 digits in a standardized test setting". How do you game this metric? The fact that one company used a couple of bad objective metrics doesn't mean all objective metrics are bad. They are used with a great deal of success in many fields. Sales people are paid on commission, traders are paid proportionally to (…

A few come to mind: cheating (see Atlanta Journal-Constitution series on this) encouraging absenteeism amongst low performers damaging the test materials of low performing students to get their scores excluded getting low performers into IEPs so they don't test, take the alternate or receive accommodations sabotage other teachers in various ways vie for the most favorable students by pulling strings with the administ…

Most of these issues can be addressed simply by completely separating education and evaluation. I.e., teachers take the day off when tests are administered, some bureaucrat shows up and does it instead.

Treating damaged test materials/absent students as a score of 0 is the simplest way to prevent hacking the student body.

vie for the most favorable students by pulling strings with the administration

The most favorable students are those who will perform better than their statistical predictor. How do teachers know who those will be?

Re: The 50 million dollar lie

#116

Post by makomk »

FourthProtocol wrote: post #5
kenjackson wrote: post #2

This is a surprising blog post in that I draw the complete opposite conclusion the author does. The author seems to think that the averaging hides volatility (which it does), which leads to incorrect conclusions drawn. Whereas to me it looks like it removes visual noise to show an actual trend. In his original scatter plot, because of the big ball in the middle it's easy to handwave and say, "look a random blob" -- b…

The other thing the author does is to derive absolute findings when the author himself admits to making an assumption about how the numbers were obtained. To me, using the word "lie" in the title is therefore nothing more than click-baiting. And on a separate note - there's a lot of anti-Gates sentiment flying around lately. Whether his methods are the best or not, he's using his own money . I find it really difficul…

There's a lot of distrust for any billionare meddling in the education system, and with good reason - they tend to push faddish ideas with questionable evidence, like this one, and somehow manage to convince the media they've "solved" the problems with the education system. Then the state wastes a whole bunch of money on the latest fad.

Also, that's probably why the headline calls it a "lie". This is something of a pattern.

Re: The 50 million dollar lie

#117

Post by yummyfajitas »

patio11 wrote: post #108
yummyfajitas wrote: post #72

Earlier quoted context omitted.

I'm really surprised to see Gary Rubinstein continuing to post bad graphs which obscure the data. 8 months ago I wrote a blog post about why you should avoid scatterplots, and used his graphs as examples of what not to do. http://www.chrisstucchio.com/blog/2012/dont_use_scatterplots... He is certainly aware of my post - he even commented on it and I offered him some constructive suggestions (such as adding alpha if h…

That is common for pre-moderated comments on blogs rather than evidence of hell banning, fwiw.

My mistake. Apologies to GR if that is the case.

Re: The 50 million dollar lie

#118

Post by CamperBob2 »

WalterBright wrote: post #83
colanderman wrote: post #59

Earlier quoted context omitted.

Your claim makes no sense. If the evaluation method measures what we want teachers to do, then it, by definition, can't be "gamed": if the measurement is high, then the teacher is doing what we want. Are you claiming that there's something special about teaching that makes teacher performance much more difficult, or even impossible, to quantify? If so, what is the point of teaching if one cannot measure results? Your…

I don't think it's so easy to manipulate a classroom full of students into thinking you're a great teacher when you're not. Remember, you're in front of them several hours a day, 5 days a week.

For what it's worth, my favorite teachers were always the ones who assigned the least work.

Re: The 50 million dollar lie

#119

Post by DanBC »

yummyfajitas wrote: post #107
DanBC wrote: post #102

Earlier quoted context omitted.

> "% of students capable of multiplying/dividing numbers up to 4 digits in a standardized test setting". How do you game this metric? You use the time that you used to use for teaching reading and devote it to teaching methods for multiplying and dividing numbers up to 4 digits in a standardized test setting. You might say this isn't gaming the test, it's just a stupid way for schools to optimise their results for th…

You use the time that you used to use for teaching reading and devote it to teaching methods for multiplying and dividing numbers up to 4 digits in a standardized test setting. Clever - you caught me. I got lazy and didn't feel like typing up all the goals of a 3'rd grade education system into an HN comment, preferring only to provide a simple example and hoping that a reasonable reader would extrapolate. Mea culpa -…

> up all the goals of a 3'rd grade education system

Can all the goals of a 3rd grade education system be reduced to a purely mechanical list of stuff?

Let's try applying your example of arithmetic to reading. What are the goals? To get children to read individual words? Or to get children to read a sentence, and obtain meaning from it? If it's just to read words, do you get those words from a defined vocab? Should they all be real words, or do you include nonsense words too?

While schools may not have stopped teaching students to read they have cut out other parts of the curriculum to focus on what's being tested.

And there's a risk of a cut off point - there's a bunch of children who will read, and there's a few children who are struggling to read. Do you spend extra time and money on the few strugglers (some of whom are going to fail whatever you do), or do you concentrate on the majority (and get most of them through the test and thus look good)?

Re: The 50 million dollar lie

#120

Post by [deleted] »

spikels wrote: post #86
vy8vWJlco wrote: post #36

Earlier quoted context omitted.

A heatmap would be more revealing, but I'd eat my hat if it weren't a standard distribution (and I had a hat). It's nature (but hey, anything's possible). The scales are only slightly skewed, but we're not talking logarithms. The merit of the scales ("value-added") on the other hand, is very questionable. But the relation, whatever it is describing, is pretty weak since the ellipse isn't very irregular (it's only "sl…

Sorry don't mean to belabor this point but I think "standard distribution" (changed from standard deviation) is still incorrect. "Standard" referers to standardized parameters (i.e. mean, variance) as in "standard normal" which has a mean of zero and variance of one (and standard deviation of one). Certainly not the case here. Maybe you mean "bivariate normal with positive correlation"? http://en.wikipedia.org/wiki/M…

[deleted]
Post reply on HN