This is silly, but I find it to be more on-topic and fun than a lot of other items that I see on HN. It's really educational to see how people look at the problem.
What tool would *you* use to solve this?
51–57 of 57 posts
Re: What tool would *you* use to solve this?
#52Permutation tests! Easy to use and easy to explain.
The two examples have the same mean and median, but differ substantially in their min/max/standard deviation/median absolute deviation. Pick your spread, they differ. I got this by poking around in iPython.
Find the standard deviation of A. Reshuffle A and B together, taking half of the resulting list, calculate the standard deviation, record it and repeat that a thousand times. See that you basically never get a standard deviation as large as you did for A alone, so they meaningfully differ.
Repeat on each of the data sets, and you see that D and E are plausibly similar to A whereas C and F are not. Repeat for B to see if we're being fucked with, and lo and behold it all seems to work out, though D is a little iffy.
I used numpy and ten lines of glue code. Happy to post if there is interest.
Re: What tool would *you* use to solve this?
#53Draw distributions in Excel. B,C and F show similar, normal shape distribution
Re: What tool would *you* use to solve this?
#54I would use the Kolmogorov-Smirnov test: http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test It's in scipy.stats.ks_2samp Results: Sets| D |p-value ----------------- A,C |0.275|0.080| B,C |0.175|0.531| ----------------- A,D |0.125|0.893| B,D |0.275|0.080| ----------------- A,E |0.100|0.983| B,E |0.300|0.043| ----------------- A,F |0.300|0.043| B,F |0.100|0.983| ----------------- As far as the test goes, if D…
Based on the final selections: there is a 98.3% chance that f is b-like there is a 98.3% chance that e is a-like there is a 89.3% chance that d is a-like there is a 53.1% chance that c is b-like If these are multiplied together, it appears that there is only a 45.8% chance that they are all classified correctly?
You're asking a good question -- but you know a lot more than what you write above.
The main thing is, you know that C, D, E, and F came from either A or B. The p-values above don't account for that; they just say what's the chance, due to random fluctuation, that a sample could have come from from the same source as A.
That's reflected in the fact that the pairs of p-values don't add to one! (Like (A,C) and (B,C) in the table above.)
You also implicitly know that at least one of {C,D,E,F} is A-like and one is B-like (otherwise there would not be a problem). So even if you know P(X and Y have same source) for all (X,Y), which you don't, you couldn't multiply them.
Finally, the p-value returned by the KS test will underestimate the true probability of discrepancy. This is because it's only looking at one thing, the max value of a CDF difference. The significant differences between the distributions may lie elsewhere, like in the tails, and the KS test is known to be relatively insensitive to tail behavior. (Although at n=40 you won't be able to see far into the tails.)
There are a host of other tests that use the same idea (empirical CDF difference) but weight differently. Some can be more effective than the KS test if you're looking for certain types of difference. Here's an OK overview, albeit for the goal of assessing normality:
http://www.instatmy.org.my/downloads/e-jurnal%202/3.pdf
In a real problem, it's always a good idea to use more empirical-cdf tests than just the KS test, to compare variances and other moments as some people in the thread have done, and to make histogram or CDF plots -- especially if you're in just 1 dimension and the plots are easy to interpret.
Re: What tool would *you* use to solve this?
#55Bash for i in {1..6}; do COL='$'$i; awk -F, "{delta = $COL - avg; avg += delta / NR; mean2 += delta * ($COL - avg); } END { print sqrt(mean2 / NR); }" list.csv; done [troll]
Troll... or hero?
haha thanks, but definitely trolling. This is perhaps the most inelegant solution in the world. But it does the job in this case. Output is 10.6014, 4.44403, 5.71353, 9.46718, 10.6471, 5.12933, for columns A to F, respectively
Re: What tool would *you* use to solve this?
#56Earlier quoted context omitted.
use R #copy data into a txt file without the first two lines from the xls par(mfrow = c(3,2)) dat for(let in c("A","B","C","D","E","F")) plot(density(dat[,let]), main = let) easy...(why doesnt HN recognise newlines?)
why doesnt HN recognise newlines? It treats single newlines as the same paragraph and double newlines as a new one. Alternatively, if you want to post code, you can indent it by four spaces. Like this: par(mfrow = c(3,2)) dat
Thanks Dove. I probably should read the posting FAQ or something.
Re: What tool would *you* use to solve this?
#57Earlier quoted context omitted.
why doesnt HN recognise newlines? It treats single newlines as the same paragraph and double newlines as a new one. Alternatively, if you want to post code, you can indent it by four spaces. Like this: par(mfrow = c(3,2)) dat
Thanks Dove. I probably should read the posting FAQ or something.
Not your fault, actually. That particular quirk may be written down somewhere, but I've not found it in my three years here.
I offer you knowledge obtained by sheer trial and error. :)