Live data from Hacker News

The 50 million dollar lie

garyrubinstein.teachforus.org

121–130 of 159 posts

Re: The 50 million dollar lie

#121

Earlier quoted context omitted.

There are quite a few statistics reported in the Gates Foundation publication. Take a look for yourself: http://www.metproject.org/downloads/MET_Validating_Using_Ran...

I never said they don't report "quite a few statistics". I said they don't report this one specific particularly important statistic, and instead obscure it behind a ridiculous averaging procedure. This was either done because they are incompetent or because they are purposely trying to mislead. That's the whole point of TFA.

What one particular statistic do you want?

Take a look at the FULL report. All the statistics behind the graph you criticize are in Table 1 on page 10.

The next 40 pages seem to be a pretty competent description of both the strengths and weaknesses of their study.

Re: The 50 million dollar lie

#122

Earlier quoted context omitted.

Or are you saying that you want to pick who to fire and who to promote based on a metric that we objectively know has only a very weak correlation with reality? What alternatives do we have? Do you know of a more accurate metric?

One alternative is to not use these flawed/complicated/expensive metrics. Imagine if I came up with a metric that had an 0.1 correlation coefficient with your performance at your job. If you're a programmer, maybe that'd be something like measuring the lines of code written per day. Would you put much faith in that? Of course not, there would be myriad problems with such a system. Same thing here. The status quo in t…

Conspicuously missing from the various weighting schemes they compare is one with 100% classroom observations.

It's not that conspicuously absent as they don't use 100% weighting for anything. That said, they do attempt to maximize ability to predict state test scores, and in that optimization classroom observation plays the smallest role of the three metrics (2-9%).

Given they appear to have to done the analysis across all weightings, I seriously doubt they'd hide this data.

I do find it odd that teachers would rather be judged based on one or two people observing their classroom and ignoring actual student output, rather than raw numbers.

As a developer I'd much rather be evaluated on some metric I could optimize for (I think some metric measuring feature value/bugs/fix rate/etc...), rather than just my manager watching me code/debug a couple of times per month.

Re: The 50 million dollar lie

#123

I suspect there'll be a lot of differing opinions on this, and I'm looking forward to seeing the discussion. What would be helpful would be if people could say how much experience they have in hard statistics, and how much what they say is driven a priori from the data. I know that hackers, in particular, have real problems with "Argument from Authority", but stats is one place where it's really, really easy to go wr…

How much formal training does the author of the blog post have? From the short bio, it doesn't sound like much.

That said, my training -- read Friedman's intro text in high school (just freetime reading on my own -- so not formal per se). Two quarters of stats as an undergrad. Two quarters in grad school. All of which at least 15 years ago -- so mostly forgotten anyways. :-)

Re: The 50 million dollar lie

#124
post #119

Earlier quoted context omitted.

You use the time that you used to use for teaching reading and devote it to teaching methods for multiplying and dividing numbers up to 4 digits in a standardized test setting. Clever - you caught me. I got lazy and didn't feel like typing up all the goals of a 3'rd grade education system into an HN comment, preferring only to provide a simple example and hoping that a reasonable reader would extrapolate. Mea culpa -…

> up all the goals of a 3'rd grade education system Can all the goals of a 3rd grade education system be reduced to a purely mechanical list of stuff? Let's try applying your example of arithmetic to reading. What are the goals? To get children to read individual words? Or to get children to read a sentence, and obtain meaning from it? If it's just to read words, do you get those words from a defined vocab? Should th…

Can all the goals of a 3rd grade education system be reduced to a purely mechanical list of stuff?

Yes, I would hope that a multi-million dollar enterprise can clearly define their goals.

What are the goals? To get children to read individual words? Or to get children to read a sentence, and obtain meaning from it?

I don't know off the top of my head whether the latter should be learned by 3rd grade. Ultimately setting the goals of our educational system is up to the various bureaucrats in the school system.

However, regardless of what the goal is, you still haven't given an way to game the system apart from "teach kids to read [words/sentences]".

While schools may not have stopped teaching students to read they have cut out other parts of the curriculum to focus on what's being tested.

Indeed - if the school system is not achieving their primary goals, they should cut secondary goals and focus on the primary ones. That's a good thing.

Do you spend extra time and money on the few strugglers (some of whom are going to fail whatever you do), or do you concentrate on the majority (and get most of them through the test and thus look good)?

This depends on what the goals of the school system are. If you want to ignore the strugglers, set the goal to be maximizing this function:

    student_scores.max() 
If you want to help the strugglers and ignore the strivers, choose this one:

    student_scores.min() 
Choose this one if helping a struggler is equally important with helping a striver:

    student_scores.mean()
This one is like the previous, but strugglers get a bit of extra weight:

    log(student_scores).mean()
Setting a goal merely forces you to acknowledge possible tradeoffs and decide which ones should be made (if the need arises).

Re: The 50 million dollar lie

#125

Earlier quoted context omitted.

One alternative is to not use these flawed/complicated/expensive metrics. Imagine if I came up with a metric that had an 0.1 correlation coefficient with your performance at your job. If you're a programmer, maybe that'd be something like measuring the lines of code written per day. Would you put much faith in that? Of course not, there would be myriad problems with such a system. Same thing here. The status quo in t…

Conspicuously missing from the various weighting schemes they compare is one with 100% classroom observations. It's not that conspicuously absent as they don't use 100% weighting for anything. That said, they do attempt to maximize ability to predict state test scores, and in that optimization classroom observation plays the smallest role of the three metrics (2-9%). Given they appear to have to done the analysis acr…

I do find it odd that teachers would rather be judged based on one or two people observing their classroom and ignoring actual student output, rather than raw numbers.

As a developer I'd much rather be evaluated on some metric I could optimize for (I think some metric measuring feature value/bugs/fix rate/etc...), rather than just my manager watching me code/debug a couple of times per month.

Okay, I really don't understand this mindset. You'd really prefer it is, say, 1/3 of your salary was determined by the number of lines of code you wrote? Or the number of bugs you closed? Or some equally inane metric that only loosely correlates with your actual performance and can be easily gamed?

What am I missing?

Re: The 50 million dollar lie

#126

Earlier quoted context omitted.

One alternative is to not use these flawed/complicated/expensive metrics. Imagine if I came up with a metric that had an 0.1 correlation coefficient with your performance at your job. If you're a programmer, maybe that'd be something like measuring the lines of code written per day. Would you put much faith in that? Of course not, there would be myriad problems with such a system. Same thing here. The status quo in t…

One alternative is to not use these flawed/complicated/expensive metrics. So replace a metric with 0.1 correlation with one having 0.0 correlation? If you are advocating that we should use the school principal's opinion rather than VAM, why do you believe opinion is superior? Do you have evidence that principal's opinion has a higher correlation with student outcomes than VAM?

So replace a metric with 0.1 correlation with one having 0.0 correlation?

Two problems with this:

1. The other metric almost certainly doesn't have an 0.0 correlation, but we don't know that for sure since the data wasn't released for whatever reason.

2. The metric with 0.1 correlation (or whatever the number is)... keep in mind the context. What is the correlation with? Test scores, something that can be and is often rigged, and is at best tangentially related to the ultimate goal which is inherently qualitative in nature. I think the comparison to programming is instructive. If you had 1/3 of your salary determined by the LOC you wrote or the number of bugs you closed, you would game the system and maximize your salary, even if you weren't providing value by doing that and even if those naive metrics did very weakly correlate with performance in some big study prior to implementing the monetary incentive. What would be gained in that scenario?

If you are advocating that we should use the school principal's opinion rather than VAM, why do you believe opinion is superior? Do you have evidence that principal's opinion has a higher correlation with student outcomes than VAM?

One of the points made in the blog I linked to is that it's unfortunate that they didn't release that data. And as I said above, it's not straightforward to assume that these weak correlations are actually meaningful in practice.

Re: The 50 million dollar lie

#127
post #119

Earlier quoted context omitted.

> up all the goals of a 3'rd grade education system Can all the goals of a 3rd grade education system be reduced to a purely mechanical list of stuff? Let's try applying your example of arithmetic to reading. What are the goals? To get children to read individual words? Or to get children to read a sentence, and obtain meaning from it? If it's just to read words, do you get those words from a defined vocab? Should th…

Can all the goals of a 3rd grade education system be reduced to a purely mechanical list of stuff? Yes, I would hope that a multi-million dollar enterprise can clearly define their goals. What are the goals? To get children to read individual words? Or to get children to read a sentence, and obtain meaning from it? I don't know off the top of my head whether the latter should be learned by 3rd grade. Ultimately setti…

> However, regardless of what the goal is, you still haven't given an way to game the system apart from "teach kids to read [words/sentences]".

Sure I have. You ignore everything that is not tested. This gets you children that pass the tests. But it ignores all the other work that schools should be doing, and it reduces education to the worst, least inspiring, mechanical drudge work.

> I don't know off the top of my head whether the latter should be learned by 3rd grade. Ultimately setting the goals of our educational system is up to the various bureaucrats in the school system.

They can't agree. That's why I listed nonsense words in one of the requirements. There's an argument about whether phonics or whole-word approaches are better, even though we have good research showing that phonics is better. And so when you look at phonics methods (include nonsense words in the tests), you get disagreement in the phonics camp, and you also have all the non-phonics people piling on.

But this should be easy to discover, right? We have millions of children learning to read each year. We randomise them, we set up a control group, we give other groups different methods. Then we test. (assuming we can get agreement on what and how to test.)

I can't imagine any politician agreeing to that.

Re: The 50 million dollar lie

#128

Earlier quoted context omitted.

You can't specify everything in the rules. For example, I know a case where management decided to hand out bonuses to the staff for minimizing inventory, because inventory costs money. The staff minimized inventory, all right, and got their bonuses. Meanwhile, the production line would regularly get halted because they'd run out of things like 5 cent resistors. It was a disaster. The same company decided to rate prog…

For teachers, the metric is (roughly speaking) "% of students capable of multiplying/dividing numbers up to 4 digits in a standardized test setting". How do you game this metric? The fact that one company used a couple of bad objective metrics doesn't mean all objective metrics are bad. They are used with a great deal of success in many fields. Sales people are paid on commission, traders are paid proportionally to (…

One popular way of gaming the metric is systematic cheating on the tests by the teachers. I say popular because it has happened on a large scale, most recently in Atlanta as I recall.

Another way to manipulate the test results is to manipulate which students are in your class or your school.

The point is, people are endlessly creative in subverting rules to their own benefit - so they conform to the letter of the rules but not the spirit.

Consider also the "work to rule" technique used by unions as a bargaining tactic. It's as simple as the workers literally adhering to their job descriptions. It doesn't work out well for the company.

Re: The 50 million dollar lie

#129

Earlier quoted context omitted.

I don't think it's so easy to manipulate a classroom full of students into thinking you're a great teacher when you're not. Remember, you're in front of them several hours a day, 5 days a week.

For what it's worth, my favorite teachers were always the ones who assigned the least work.

For some students, sure, but a majority? But I'm not so sure that will impress either the parents or the staff. Also, was that work busywork, or productive work? Students can tell the difference.

I had a high school teacher who explicitly wanted to be popular by assigning no work. The students did like him, but they did not respect him (and laughed at him behind his back). Those of us who wanted to go on to college also knew he was shortchanging us.

Re: The 50 million dollar lie

#130
post #7

I know nothing about the domain of teacher measurement, and have no opinion on it. But the first chart in the blog post, with the big mass of blue on it, is surely not the strong evidence of weak correlation, that the author is making it out to be? There could still be a strong correlation in that data, even if it looks like a blob of blue - because there are so many data points on that chart, that we can no longer t…

Important expectation-calibration note: I'm not making any claims about teacher performance, the usefulness of the value-added idea, or any lofty topics in general. I'm just trying to figure out how to recreate the first graph in the initial posting and get a closer look at the data behind it. I grabbed the data set [1], loaded it into R, and tried to recreate the first graph in the original post. Observations: 1. Th…

One question would be whether or not to match the files on subject/grade/teacher, or just the teacher.

You use subject/grade/teacher level, but you get a different answer if you use aggregated teacher level.

Post reply on HN