Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

zedshaw.com

Post reply on HN
Print view View original

41–50 of 122 posts

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#41

Post by duncanj »

stcredzero wrote: post #21
dschobel wrote: post #15

Serious question: who here who has at least a BS in CS didn't have a mandatory stats class where you learned all about picking sample sizes to give you an accuracy you're happy with or that an average without a standard deviation is nigh-meaningless? It's just a fact of our profession, there is a significant percentage of people who just slap together APIs and have zero understanding of the maths behind it. I don't s…

There's a big percentage of programmers who don't understand O(n^2). On multiple occasions, I've seemed as a guru because of this one little tidbit. What if the medical profession has as much egregious and widespread ignorance of the basics as programming? Would you be in favor of certification?

In medical professions, you go through a period of training (clinicals, internship, residency) where people with tons of experience point out that you never learned anything. This humbling experience is not really available in the CS world.

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#42

Post by awad »

travisjeffery wrote: post #35
stcredzero wrote: post #30

Earlier quoted context omitted.

Reminds me of the film clip Alan Kay showed at one of his talks. It was taken at a Harvard graduation, and it shows many, many Harvard graduates, students, and faculty saying that the Earth was warmer in Summer because the orbit takes it closer to the sun. http://www.sigchi.org/chi98/chikids/vol2/headlines/head2.htm...

It's caused by tilt/angle of Earth, correct?

yes, since we're on a tilt the two hemispheres get their summer and winter during opposite seasons when their respective hemisphere is 'facing' the sun while the equator pretty much always gets the same amount of sun

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#43

Post by stcredzero »

awad wrote: post #42
travisjeffery wrote: post #35

Earlier quoted context omitted.

It's caused by tilt/angle of Earth, correct?

yes, since we're on a tilt the two hemispheres get their summer and winter during opposite seasons when their respective hemisphere is 'facing' the sun while the equator pretty much always gets the same amount of sun

One of the things that happens in the film clip, is that a professorial gentleman gives the "orbit is closer" answer, then is asked what season it is in the Southern hemisphere when it's Summer in the Northern one.

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#44

Post by stcredzero »

duncanj wrote: post #41
stcredzero wrote: post #21

Earlier quoted context omitted.

There's a big percentage of programmers who don't understand O(n^2). On multiple occasions, I've seemed as a guru because of this one little tidbit. What if the medical profession has as much egregious and widespread ignorance of the basics as programming? Would you be in favor of certification?

In medical professions, you go through a period of training (clinicals, internship, residency) where people with tons of experience point out that you never learned anything. This humbling experience is not really available in the CS world.

Particularly disturbing, since the only good way I know of to learn programming and system architecture is through mentoring. Basically, our mentoring system is haphazard. There's even a lot of anti-mentoring happening out there.

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#45

Post by baguasquirrel »

dschobel wrote: post #15

Serious question: who here who has at least a BS in CS didn't have a mandatory stats class where you learned all about picking sample sizes to give you an accuracy you're happy with or that an average without a standard deviation is nigh-meaningless? It's just a fact of our profession, there is a significant percentage of people who just slap together APIs and have zero understanding of the maths behind it. I don't s…

At CMU we had a mandatory stats class but it didn't have anything concerning sample size picking. Ironically, the stats class I had to take when I was in the humanities dept. was more comprehensive. When I walked into industry, I realized after two or three years that practically everyone I'd met knew jack shit about statistics. I myself learned most of what I know from humanities-major friends and books like NLP by Manning and Schutze.

Having been through what Zed's been through (on both the giving and receiving end to wit), I can empathize with why he'd get all sanctimonious about it, but I feel that it just makes the problem worse (at least in person). Programmers are already an egotistical lot, and I've learned that directly attacking their ego tends to make things worse.

What can be done then? I don't know. It feels like this is at the core a personality problem, in particular one in which people associate their ability to know with their sense of self, and personality problems are terribly hard to correct.

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#46

Post by stcredzero »

m_ wrote: post #31
megaduck wrote: post #23

Earlier quoted context omitted.

The problem isn't that people know nothing about statistics. The problem is that they think they know more than they actually do. It's fairly easy to get past your "intro to stats" class with only a passing familiarity with the subject and no deep understanding. Only once you truly grasp stats do you realize that you don't actually know anything. Any statistical formula will predict its own failure. That's kind of mi…

I studied CS and I got past my "intro to stats" class with only a passing familiarity with the subject and no deep understanding (aside from the probability aspects of the course). The professor was very poor overall. I was able to cram enough to do well, but I retained very little after the final. I would love a deep understanding built from the foundations-- the coursework we had was a lot more obtuse and "take my…

Why aren't there statistics for CS classes that put stats primarily in the context of system profiling and metrics? Are there any out there?

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#47

Post by nkurz »

I don't like this article. Yes, he's right: an intuitive knowledge of the relationship between averages and standard deviations is essential. But to presume that the standard tools of statistics should be applied to every problem is to miss the point.

On a superficial level, if you are doing overnight processing of log files, then you probably care more about throughput than latency. In this case, averages are probably a fine metric. On a slightly deeper level, standard deviation is only a useful measure if the distribution is known, and in a lot of real world cases it is not. The right question isn't whether 100 or 1000 tests on the same data provides sufficient statistical power, but whether range of inputs is sufficient to trigger worst case perfomance.

Now, I presume that Zed knows these things and applies them appropriately, but the article strikes me as more snide than helpful. Perhaps as others say he's a great guy in person, but I prefer my stats with less attitude and more insight. Here, for example: http://yudkowsky.net/rational/bayes

[edit: changed my sloppy language from 'has no meaning unless to the distribution is normal' to 'is only a useful measure if the distribution is known']

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#48

Post by tokenadult »

travisjeffery wrote: post #35
stcredzero wrote: post #30

Earlier quoted context omitted.

Reminds me of the film clip Alan Kay showed at one of his talks. It was taken at a Harvard graduation, and it shows many, many Harvard graduates, students, and faculty saying that the Earth was warmer in Summer because the orbit takes it closer to the sun. http://www.sigchi.org/chi98/chikids/vol2/headlines/head2.htm...

It's caused by tilt/angle of Earth, correct?

It's caused by tilt/angle of Earth, correct?

Yes. My second-grade teacher made sure her class knew this. A year or two later I learned that the earth is closer to the sun (at perihelion) when the SOUTHERN Hemisphere is having its summer, and the Northern Hemisphere is having its winter.

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#49

Post by tokenadult »

nkurz wrote: post #47

I don't like this article. Yes, he's right: an intuitive knowledge of the relationship between averages and standard deviations is essential. But to presume that the standard tools of statistics should be applied to every problem is to miss the point. On a superficial level, if you are doing overnight processing of log files, then you probably care more about throughput than latency. In this case, averages are probab…

On a slightly deeper level, standard deviation only has meaning if the distribution is presumed to be normal

Are you completely sure about that?

I suppose many readers of this thread are more knowledgeable about statistics than I am. I would appreciate hearing from the knowledgeable readers whether or not variance in the observed values makes a difference in the cases discussed in the submitted article.

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#50

Post by WilliamLP »

> I think women are better programmers because they have less ego and are typically more interested in the gear rather than the pissing contest.

People need to stop writing shit like this. If it's not okay to say that men are better programmers simply because of their gender (it's not), then it's not okay to say it the other way around either.

Post reply on HN