Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

zedshaw.com

Post reply on HN
Print view View original

51–60 of 122 posts

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#51

Post by jerf »

edu wrote: post #11

So after Ruby and Ruby on Rails, Zed has learnt some R and statistics. Because after reading almost all the article, he basically talks about: average, median, standard deviation. He should learn a little bit of complexity and calculability by the way, best case, worst case, Big O notation, etc.

Because when edu writes a blog post, he includes his entire life history, and, for that matter, the entire history of the universe in it.

I know you've already been smacked down but this is a big pet peeve of mine and worth pointing out. You can always complain that something should have had more information in it, and since that is always true no matter what, the complaint is information free. (A value that comes from a universe of one value contributes zero bits of information.)

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#52

Post by cschep »

stcredzero wrote: post #46
m_ wrote: post #31

Earlier quoted context omitted.

I studied CS and I got past my "intro to stats" class with only a passing familiarity with the subject and no deep understanding (aside from the probability aspects of the course). The professor was very poor overall. I was able to cram enough to do well, but I retained very little after the final. I would love a deep understanding built from the foundations-- the coursework we had was a lot more obtuse and "take my…

Why aren't there statistics for CS classes that put stats primarily in the context of system profiling and metrics? Are there any out there?

That would have been cool. Stats wasn't required for us, but if it would have been Statistical Analysis of Computer Systems(or programs, or whatever), I would have signed up for it anyway.

I heard the stats class offered where I went to school was a joke so I didn't pursue it optionally. I'm sure it's different depending on your university.

Also, my school was not renowned for any sciences.

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#53

Post by jfarmer »

That's a lot of text to say not much more than "Standard deviation can be as important as the mean; be careful about confounding variables; and if you're an engineer, spend more time learning statistics."

Oh, also, "You're all assholes and I rock."

Truly the Carl Sagan of software.

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#54

Post by sp332 »

callmeed wrote: post #38

Can anyone recommend a decent (read "not boring") book on statistics? Something along the lines of "Naked Economics" for the stats realm ...

There's the Cartoon Guide to Statistics ( http://www.amazon.com/Cartoon-Guide-Statistics-Larry-Gonick/... ), which, despite its name, is pretty solid and comprehensive book on basics. And definitely not boring!

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#56

Post by nazgulnarsil »

tokenadult wrote: post #48
travisjeffery wrote: post #35

Earlier quoted context omitted.

It's caused by tilt/angle of Earth, correct?

It's caused by tilt/angle of Earth, correct? Yes. My second-grade teacher made sure her class knew this. A year or two later I learned that the earth is closer to the sun (at perihelion) when the SOUTHERN Hemisphere is having its summer, and the Northern Hemisphere is having its winter.

which is why the northern hemisphere has a slightly milder climate.

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#57

Post by nazgulnarsil »

msluyter wrote: post #17

I hadn't seen this, so I was glad it was resubmitted. I think he has a fair point. Here on HN I see a lot of armchair sociologists critique the various articles in the social sciences that get posted, but it's rather unclear to me whether these are well grounded in an actual understanding of the issues involved, or simply habitual incantations of rules of thumb such as "correlation doesn't imply causation."

um...rules of thumb are a bit different from identifying logical fallacies.

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#58

Post by nkurz »

tokenadult wrote: post #49
nkurz wrote: post #47

I don't like this article. Yes, he's right: an intuitive knowledge of the relationship between averages and standard deviations is essential. But to presume that the standard tools of statistics should be applied to every problem is to miss the point. On a superficial level, if you are doing overnight processing of log files, then you probably care more about throughput than latency. In this case, averages are probab…

On a slightly deeper level, standard deviation only has meaning if the distribution is presumed to be normal Are you completely sure about that? I suppose many readers of this thread are more knowledgeable about statistics than I am. I would appreciate hearing from the knowledgeable readers whether or not variance in the observed values makes a difference in the cases discussed in the submitted article.

Are you completely sure about that?

You could quibble about my exact phrasing, but yes, I'm completely sure about that. This is the 'black swan' problem writ small. I don't mean that a high standard deviation should be ignored for real-world distributions, but I do mean that a low standard deviation carries very little weight unless a normal distribution is presumed.

I'm hard pressed to relate this to the cases discussed in the article, as those cases are shy on detail, but the DB2 example seems most applicable. Although he points to standard deviation as the tell-tale flag here, this is sort of misleading. The exact numerical value for the standard deviation across all queries is meaningless here, as not every query has an equal likelihood of being slow. As he states, the real problem was the terrible performance of an single query.

How many similar queries exist? Will a new query added to the system trigger a similar bug? We don't know, and standard statistics isn't going to help us unless we have an understanding of the underlying mechanism. The key here is not to test a statistically significant subset of all possible queries, but to check the performance of the actual queries executed (as he did).

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#59

Post by stcredzero »

cschep wrote: post #52
stcredzero wrote: post #46

Earlier quoted context omitted.

Why aren't there statistics for CS classes that put stats primarily in the context of system profiling and metrics? Are there any out there?

That would have been cool. Stats wasn't required for us, but if it would have been Statistical Analysis of Computer Systems(or programs, or whatever), I would have signed up for it anyway. I heard the stats class offered where I went to school was a joke so I didn't pursue it optionally. I'm sure it's different depending on your university. Also, my school was not renowned for any sciences.

A really good textbook along these lines might really be something.

Re: Programmers Need To Learn Statistics Or I Will Kill Them All (2005)

#60

Post by nailer »

stcredzero wrote: post #32
biohacker42 wrote: post #27

Earlier quoted context omitted.

I wonder if this is a "feature" of women or all minorities? For example, do male nurses show the same egolessness as female programmers?

As a minority Irish musician, I find that this seems to have a dampening effect on my ego. (Not 100% sufficient to keep me from making a fool of myself occasionally, though.) It's also caused me to be very, very paranoid about whether or not I "get it" and as a result I've had bouts of beneficial studiousness. And yes, I have to do better to win respect, and sometimes I am judged by my skin color, not my actual playi…

I thought Irish people were renowned for their musicality. Eg, singing is a common social activity in Ireland (to the point where it's socially unacceptable not to sing), and Ireland has produced far more commercially successful musicians than their population would suggest. What country are you playing in?
Post reply on HN