Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

zedshaw.com

Post reply on HN
Print view View original

111–120 of 122 posts

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#111

Post by tjic »

ilitirit wrote: post #109
tjic wrote: post #94

Earlier quoted context omitted.

> If it's not okay to say that men are better programmers simply because of their gender (it's not), then it's not okay to say it the other way around either. God forbid that someone make an argument and try to support it with data, when one person has already decided for all of us what the correct conclusion is. WT* ever happened to debate, intellectual discourse, and the marketplace of ideas? This is one of the thi…

> God forbid that someone make an argument and try to support it with data That's well and fine, but the author stated an opinion based on anecdotal evidence. On its own though, I think it's a really stupid thing to say but I think it fits fine given the tone of his rant. While I do agree with your central point, I don't think there's any value in exploring whether or not women might make better programmers on averag…

> If I were to hire someone, I'd base it on their past experience and how well the interview(s) went, not their race or sex.

Absolutely. I agree 100%.

OTOH, discussing things in aggregates also makes sense.

If 4 out of 1,000 women would make excellent engineers, and 1 out of 1,000 men would make excellent engineers, and yet we see that the distribution of actual engineers is something other than 4:1, we should investigate.

If, OTOH, the numbers are 1 out of 1,000 and 10 out of 1,000 respectively, and the ratio of actual engineers is 1:10, then we might choose to spend less time and energy on the investigation.

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#112

Post by likpok »

mr_justin wrote: post #71
m_ wrote: post #31

Earlier quoted context omitted.

I studied CS and I got past my "intro to stats" class with only a passing familiarity with the subject and no deep understanding (aside from the probability aspects of the course). The professor was very poor overall. I was able to cram enough to do well, but I retained very little after the final. I would love a deep understanding built from the foundations-- the coursework we had was a lot more obtuse and "take my…

By "intro to stats" do you mean the calc-based stats class that most people in CS take? I'd hardly call that an intro class considering you need to complete Calc I & II before enrolling.

Like many things, Stats is a vast field. Would you call a class about the basics of groups and rings anything other than an intro to abstract algebra, even though most places require both calculus and a fair amount of mathematical background?

This is sort of Zed's point. It is an intro class into a field with a great deal of depth. All you can do in one undergraduate semester is touch on a few of the most important parts.

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#114

Post by ilitirit »

tjic wrote: post #111
ilitirit wrote: post #109

Earlier quoted context omitted.

> God forbid that someone make an argument and try to support it with data That's well and fine, but the author stated an opinion based on anecdotal evidence. On its own though, I think it's a really stupid thing to say but I think it fits fine given the tone of his rant. While I do agree with your central point, I don't think there's any value in exploring whether or not women might make better programmers on averag…

> If I were to hire someone, I'd base it on their past experience and how well the interview(s) went, not their race or sex. Absolutely. I agree 100%. OTOH, discussing things in aggregates also makes sense. If 4 out of 1,000 women would make excellent engineers, and 1 out of 1,000 men would make excellent engineers, and yet we see that the distribution of actual engineers is something other than 4:1, we should invest…

Maybe it's because you're using a contrived example, but I don't quite agree. These sort of studies are usually undertaken at a higher, more abstract level. eg. How do various cognitive abilities differ between sexes? With the resulting data, we might be able to explain certain phenomena, or dispel certain stereotypes. But I don't think we're anywhere close to being in a position we can say x out y men/women would make excellent engineers. That kind of information is usually extracted from trend analyses (eg. studying the proportion of males/female engineers who are highly successful at their career vs those who aren't), and they're inherently skewed because of the nature of the society we live in.

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#115

Post by randallsquared »

tjic wrote: post #111
ilitirit wrote: post #109

Earlier quoted context omitted.

> God forbid that someone make an argument and try to support it with data That's well and fine, but the author stated an opinion based on anecdotal evidence. On its own though, I think it's a really stupid thing to say but I think it fits fine given the tone of his rant. While I do agree with your central point, I don't think there's any value in exploring whether or not women might make better programmers on averag…

> If I were to hire someone, I'd base it on their past experience and how well the interview(s) went, not their race or sex. Absolutely. I agree 100%. OTOH, discussing things in aggregates also makes sense. If 4 out of 1,000 women would make excellent engineers, and 1 out of 1,000 men would make excellent engineers, and yet we see that the distribution of actual engineers is something other than 4:1, we should invest…

If 4 out of 1,000 women would make excellent engineers, and 1 out of 1,000 men would make excellent engineers, and yet we see that the distribution of actual engineers is something other than 4:1, we should investigate.

We might want to investigate, but "should" is a strong word. There are all kinds of reasons why this could happen, and some of them might not need fixing.

It could be that the total number of people who would make excellent engineers is too low for the number needed, and that men (in your example) are more likely to make adequate or good engineers than women, even though women are more likely to make excellent ones.

It could be that women in general, in spite of being four times as likely to make excellent engineers, tend not to enjoy engineering for cultural or other reasons.

It could be that the women who would be excellent engineers are in the group that would be pretty good CEOs, and they all become CEOs because there's more money in it.

But you get my point: the failure of actual people to conform to the occupations that they would be best at is not, in and of itself, evidence of a problem.

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#116

Post by randallsquared »

ironkeith wrote: post #84
GHFigs wrote: post #79

Earlier quoted context omitted.

phone in your job I know the talk you're referring to and he did not say this. He does suggest doing your best and working really hard to make sure your employers get what they want. Do the job you're paid to do and do it well. What he advises against is going the extra mile for a company that does not trust your judgement and that you have no stake in. Leave work at work. Your job is not your life. You don't owe the…

You'll forgive my hyperbole, but I was trying to make a point. What I'm trying to get at is the reason the people in charge of most companies are "clueless about technology" is that some technical people aren't capable of speaking to them in their language. The people in charge are going to be more concerned with how these issues affect their bottom line, if you can't communicate in those terms you'll rarely be effec…

the reason the people in charge of most companies are "clueless about technology" is that some technical people aren't capable of speaking to them in their language.

The reason I'm clueless about marketing is not that marketing people aren't capable of speaking to me in my language. It's that, fundamentally, I don't care about marketing. It seems likely to me that the people in charge of most companies really don't care about technology (nor should they except in cases it's critical for the company; I'm not criticizing).

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#118

Post by GHFigs »

ironkeith wrote: post #84
GHFigs wrote: post #79

Earlier quoted context omitted.

phone in your job I know the talk you're referring to and he did not say this. He does suggest doing your best and working really hard to make sure your employers get what they want. Do the job you're paid to do and do it well. What he advises against is going the extra mile for a company that does not trust your judgement and that you have no stake in. Leave work at work. Your job is not your life. You don't owe the…

You'll forgive my hyperbole, but I was trying to make a point. What I'm trying to get at is the reason the people in charge of most companies are "clueless about technology" is that some technical people aren't capable of speaking to them in their language. The people in charge are going to be more concerned with how these issues affect their bottom line, if you can't communicate in those terms you'll rarely be effec…

but he was unable to convince the powers that be that he was right

Yes he was. That was the point of him giving advice on the subject. It wouldn't make much sense to suggest a bunch of things that didn't work at all, would it?

Zed basically abdicates responsibility for having to communicate with non-technical people

He presents a six-point strategy for doing exactly that. I suggest watching the talk again, as you seem to have missed out on most of it.

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#119

Post by lutorm »

voberoi wrote: post #105
dschobel wrote: post #15

Serious question: who here who has at least a BS in CS didn't have a mandatory stats class where you learned all about picking sample sizes to give you an accuracy you're happy with or that an average without a standard deviation is nigh-meaningless? It's just a fact of our profession, there is a significant percentage of people who just slap together APIs and have zero understanding of the maths behind it. I don't s…

I didn't, and I'm graduating with a BS in CS from Stanford in a few weeks. We have an introductory stats requirement, but I found that the stats class I took my sophomore year for my (dropped) economics major was far more enlightening than my CS stats requirement. I think the fundamental issue is that there's very little focus on statistical practice in most of these courses. Social scientists have it good: they're a…

For a different view of how useful statistical practice (when applied mindlessly) is, read "The Black Swan". It's all about how people use statistical models with Normal distributions in places where it's patently unjustified, and the price we pay for it.

I played this game myself with a friend. I sent him ten samples of a (for him) unknown distribution and asked him to estimate the mean. Then 100, then 1000. His estimate of the mean kept changing to higher and higher values, because the samples were drawn from a Pareto (power-law) distribution with a mean of 1000. Such a distribution is almost indistinguishable from one with a mean of infinity, because all the signal is in the very rare, large outliers. If you try to analyze samples from such a process assuming it's Gaussian, nothing will make sense, and the standard deviation will give you an estimated uncertainty of the mean that is far, far below the actual uncertainty.

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#120

Post by philwelch »

tjic wrote: post #95
philwelch wrote: post #82

Earlier quoted context omitted.

It's bad usage, but in sociological terms, women are a minority because they aren't the dominant subgroup.

> in sociological terms, women are a minority because they aren't the dominant subgroup. This is a useful sentence to read, because it helps remind me that "sociological terms" have little or nothing to do with honest intellectual framing of topics.

I had largely the same reaction you did when I saw "minority" used that way.

You're right. "Minority" is a really bad word for what biohacker42 was trying to say, because many "minorities" (in terms of social status) are not numeric minorities. (Blacks in apartheid-era South Africa. Women throughout history.) As jfoutz mentioned, we both know a pointer is not a dog.

It's convenient that you misquoted me, too, because doing so leaves out the part where I already conceded that it's bad usage. Which leaves me at a loss as to what you're trying to add to the discussion--did you say something more than what I've already implied (and exhaustively reiterated here)?

Post reply on HN