Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

zedshaw.com

Post reply on HN
Print view View original

61–70 of 122 posts

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#61

Post by menloparkbum »

nailer wrote: post #60
stcredzero wrote: post #32

Earlier quoted context omitted.

As a minority Irish musician, I find that this seems to have a dampening effect on my ego. (Not 100% sufficient to keep me from making a fool of myself occasionally, though.) It's also caused me to be very, very paranoid about whether or not I "get it" and as a result I've had bouts of beneficial studiousness. And yes, I have to do better to win respect, and sometimes I am judged by my skin color, not my actual playi…

I thought Irish people were renowned for their musicality. Eg, singing is a common social activity in Ireland (to the point where it's socially unacceptable not to sing), and Ireland has produced far more commercially successful musicians than their population would suggest. What country are you playing in?

I read his sentence as that he's not Irish, but plays Irish music.

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#62

Post by ZeroGravitas »

cschep wrote: post #52
stcredzero wrote: post #46

Earlier quoted context omitted.

Why aren't there statistics for CS classes that put stats primarily in the context of system profiling and metrics? Are there any out there?

That would have been cool. Stats wasn't required for us, but if it would have been Statistical Analysis of Computer Systems(or programs, or whatever), I would have signed up for it anyway. I heard the stats class offered where I went to school was a joke so I didn't pursue it optionally. I'm sure it's different depending on your university. Also, my school was not renowned for any sciences.

I once listened to a stats professor bemoaning that he had to teach stats to various groups of students in their first year when they didn't know enough about their own subjects for him to use any relevant examples.

Since stats on it's own can be a bit abstract that's a double whammy.

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#63

Post by npk »

tokenadult wrote: post #49
nkurz wrote: post #47

I don't like this article. Yes, he's right: an intuitive knowledge of the relationship between averages and standard deviations is essential. But to presume that the standard tools of statistics should be applied to every problem is to miss the point. On a superficial level, if you are doing overnight processing of log files, then you probably care more about throughput than latency. In this case, averages are probab…

On a slightly deeper level, standard deviation only has meaning if the distribution is presumed to be normal Are you completely sure about that? I suppose many readers of this thread are more knowledgeable about statistics than I am. I would appreciate hearing from the knowledgeable readers whether or not variance in the observed values makes a difference in the cases discussed in the submitted article.

The statement "only has meaning if the distribution is presumed to be normal" is wrong. The SD is a summary of the spread of a distribution. In fact, for most centrally concentrated distributions (including a uniform one) +/- 1 sigma corresponds to about 60% of the mass of the distribution. This is an amazingly useful thing to know.

As the above triva factoid points out, the standard deviation is an important summary statistic. More interestingly by using mean, variance (or sd), skew, and kurtosis, you can describe almost any centrally concentrated distribution. Even distribution with heavy tails.

I think what the OP meant is that most 3+ sigma results are not truly 3+ sigma, because most distributions in this world are not gaussian, but instead have large wings. SD is most useful when you know what the underlying distribution is. Currently it's more in fashion to communicate spread using confidence intervals because they presume less about the underlying distribution.

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#64

Post by lutorm »

stcredzero wrote: post #30
tokenadult wrote: post #8

The article does seem familiar, and I think I've even seen it on HN before. http://news.ycombinator.com/item?id=48006 Yep. (There seems to be a slight change in the base URL for the submitted article, but this has been discussed here before.) Still this is well worth discussing again. It's amazing how much most college-educated people think they know about statistics that they really don't. Two of my favorite quick o…

Reminds me of the film clip Alan Kay showed at one of his talks. It was taken at a Harvard graduation, and it shows many, many Harvard graduates, students, and faculty saying that the Earth was warmer in Summer because the orbit takes it closer to the sun. http://www.sigchi.org/chi98/chikids/vol2/headlines/head2.htm...

That's part of a film called "The private universe" (http://www.learner.org/resources/series28.html) about how students can learn stuff in class and then promptly revert to their nonscientific understanding unless they are forced to really confront their prior knowledge in light of what they are being taught. It's a fascinating film.

The importance of explicitly confronting prior knowledge and integrating it into learning is one of the basic principles of learning that came out of the "How People Learn" studies (http://www.nap.edu/openbook.php?record_id=6160).

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#65

Post by nkurz »

npk wrote: post #63
tokenadult wrote: post #49

Earlier quoted context omitted.

On a slightly deeper level, standard deviation only has meaning if the distribution is presumed to be normal Are you completely sure about that? I suppose many readers of this thread are more knowledgeable about statistics than I am. I would appreciate hearing from the knowledgeable readers whether or not variance in the observed values makes a difference in the cases discussed in the submitted article.

The statement "only has meaning if the distribution is presumed to be normal" is wrong. The SD is a summary of the spread of a distribution. In fact, for most centrally concentrated distributions (including a uniform one) +/- 1 sigma corresponds to about 60% of the mass of the distribution. This is an amazingly useful thing to know. As the above triva factoid points out, the standard deviation is an important summary…

You're right. I was being sloppy.

I should have said something more like "the standard deviation calculated from a sample set is only generally applicable in so far as one is willing to make assumptions that the sample set is representative of the distribution as a whole". The default assumption in traditional statistics (such as quoting p-values) is that the distribution is normal, and in real world situations often not the case.

Your restatement is right on, although I'd go farther and say that standard deviations (and confidence intervals) are only useful metrics with regard to the particular assumptions one is willing to make about underlying distribution. Yes, you can calculate these measures, but they won't help you if your assumptions are irreparably flawed.

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#66

Post by stcredzero »

ZeroGravitas wrote: post #62
cschep wrote: post #52

Earlier quoted context omitted.

That would have been cool. Stats wasn't required for us, but if it would have been Statistical Analysis of Computer Systems(or programs, or whatever), I would have signed up for it anyway. I heard the stats class offered where I went to school was a joke so I didn't pursue it optionally. I'm sure it's different depending on your university. Also, my school was not renowned for any sciences.

I once listened to a stats professor bemoaning that he had to teach stats to various groups of students in their first year when they didn't know enough about their own subjects for him to use any relevant examples. Since stats on it's own can be a bit abstract that's a double whammy.

Then stats in CS should be a 2nd or 3rd year course. By then, there should be a great deal of material usable as examples.

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#67

Post by delano »

biohacker42 wrote: post #27
stcredzero wrote: post #20

Oh, and you wonder why I say, “he”? I never have this problem with female programmers...I think women are better programmers because they have less ego and are typically more interested in the gear rather than the pissing contest. When I was hanging out in Homer Alaska, squatting in the tent city on the beach, I heard that on the halibut fishing tours, the women would catch more and bigger fish. (And halibut are big…

I wonder if this is a "feature" of women or all minorities? For example, do male nurses show the same egolessness as female programmers?

I might be misunderstanding your comment, but women are not a minority.

http://www16.wolframalpha.com/input/?i=male+population+in+no...

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#68

Post by stcredzero »

menloparkbum wrote: post #61
nailer wrote: post #60

Earlier quoted context omitted.

I thought Irish people were renowned for their musicality. Eg, singing is a common social activity in Ireland (to the point where it's socially unacceptable not to sing), and Ireland has produced far more commercially successful musicians than their population would suggest. What country are you playing in?

I read his sentence as that he's not Irish, but plays Irish music.

Correct. And for some reason, I am often the only one who ever gets asked about their origins, even though some of my fellow musicians are of Cajun, Hungarian, German, Mexican, and Swedish extraction.

For many, Irish music is tied up in the context of some sort of romanticized Irish Nationalism, and is only appreciated as some sort of ethnic tchotchke. Their imagination fails at the notion that I'm enjoying it in a purely musical context. I associate those people with the ones in the audience who can't clap on the beat.

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#69

Post by stcredzero »

lutorm wrote: post #64
stcredzero wrote: post #30

Earlier quoted context omitted.

Reminds me of the film clip Alan Kay showed at one of his talks. It was taken at a Harvard graduation, and it shows many, many Harvard graduates, students, and faculty saying that the Earth was warmer in Summer because the orbit takes it closer to the sun. http://www.sigchi.org/chi98/chikids/vol2/headlines/head2.htm...

That's part of a film called "The private universe" ( http://www.learner.org/resources/series28.html ) about how students can learn stuff in class and then promptly revert to their nonscientific understanding unless they are forced to really confront their prior knowledge in light of what they are being taught. It's a fascinating film. The importance of explicitly confronting prior knowledge and integrating it into l…

Thanks for the link. I was wondering where that clip was from. Reminds me of Feynman's bit from his time teaching in Brazil. The students there could stand up in class and recite the textbook description of Polarized Light, but when asked to identify it in the real world in the view out the window, couldn't. (The definition mentioned the example of reflected light, and there was sunlight playing off of water in the bay.)

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#70

Post by DannoHung »

nazgulnarsil wrote: post #56
tokenadult wrote: post #48

Earlier quoted context omitted.

It's caused by tilt/angle of Earth, correct? Yes. My second-grade teacher made sure her class knew this. A year or two later I learned that the earth is closer to the sun (at perihelion) when the SOUTHERN Hemisphere is having its summer, and the Northern Hemisphere is having its winter.

which is why the northern hemisphere has a slightly milder climate.

Huh, is this actually true? I had been under the impression that the orbital difference was insignificant from a climate standpoint.
Post reply on HN