What tool would *you* use to solve this? • Hacker News Board

Re: What tool would you use to solve this?

#11

Post by ColinWright » Wed, Oct 05, 2011, 9:28 AM UTC

Several people have said they would, or have, ploted histograms - but how?

* By hand?

* With Excel?

* With R?

* With processing?

I'm a little sad to see that this item has been flagged heavily, but I guess there are people who think this is sufficiently off-topic that it should be in the same category as spam.

Re: What tool would you use to solve this?

#12

Post by Cieplak » Wed, Oct 05, 2011, 9:31 AM UTC

Bash

for i in {1..6}; do COL='$'$i; awk -F, "{delta = $COL - avg; avg += delta / NR; mean2 += delta * ($COL - avg); } END { print sqrt(mean2 / NR); }" list.csv; done

[troll]

Re: What tool would you use to solve this?

#13

Post by Dn_Ab » Wed, Oct 05, 2011, 10:13 AM UTC

I would look at the mean and variance in each column and cluster by that. My intuition is that the variance in Temperature measured in Fahrenheit will be greater then that of weight in Kg.

Checking it seems that this is borne out. So is the data real? What is being assessed here?

Interesting to see that people prefer std dev to variance for this problem.

Re: What tool would you use to solve this?

#14

Post by anthonyb » Wed, Oct 05, 2011, 10:20 AM UTC

I can just scan the numbers and see the patterns. Column A has much more extreme ranges, lots of 40s and 70s, whereas B is much narrower.

So D and E are temperature, and C and F are weights.

Re: What tool would you use to solve this?

#15

Post by praptak » Wed, Oct 05, 2011, 10:28 AM UTC

An interesting problem. My first guess is that teenagers' weights distribution is two-peaked (boys & girls) but this assumes that the teenagers are around the same age and that the peaks are far enough to be detectable in the samples. I can't be arsed to check this hypothesis though :-)

Anyway, a serious approach to this problem would require comparing against solid real-life data. Off the bat assumptions about distributions of real life data are often very wrong.

Re: What tool would you use to solve this?

#16

Post by revorad » Wed, Oct 05, 2011, 10:56 AM UTC

I used my app https://app.prettygraph.com to quickly look at the histograms. B and F look almost identical with a very normal distribution. The others look more like twin-peaks.

Edit: If you want to try, copy the data and click the "Paste data from Excel link" on the left under the Data Tab, paste and load. Then choose histogram under the Graph tab. It plots the X var (ignores Y). Sorry it's a bit shabby, I haven't worked on it in a while.

Re: What tool would you use to solve this?

#17

Post by [deleted] » Wed, Oct 05, 2011, 10:58 AM UTC

ColinWright wrote: post #11

Several people have said they would, or have, ploted histograms - but how? * By hand? * With Excel? * With R? * With processing? I'm a little sad to see that this item has been flagged heavily, but I guess there are people who think this is sufficiently off-topic that it should be in the same category as spam.

[deleted]

Re: What tool would you use to solve this?

#18

Post by stephen789 » Wed, Oct 05, 2011, 10:58 AM UTC

Here is the scatter plot. Sorted. The difference isn't hard to see. https://docs.google.com/spreadsheet/oimg?key=0Ar0JB_woDdlodD...

Re: What tool would you use to solve this?

#19

Post by andrewcooke » Wed, Oct 05, 2011, 11:00 AM UTC

as others have said, first thing to do is plot the data. but if you want a statistical test (with significances, so that you have some idea of how confident you can be in your decision) then you could use the two-sample ks test. it's likely more useful that a simple t-test here because it's sensitive to shape.

but to be honest the first thing i would do is google for suitable approaches...

Re: What tool would you use to solve this?

#20

Post by hackermom » Wed, Oct 05, 2011, 11:10 AM UTC

Does the thread starter wonder about what software tool (i.e. language) people would use, or does he wonder about what method people would use? As far as language goes, the problem requires mathematics on a level so simple that practically any programming language is more than enough sufficient - the histographical method I'd use for solving this problem would be doable in BASIC on a VIC-20 or a TI-81 pocket calculator.