Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: st – simple statistics from the command line

github.com

Post reply on HN
Print view View original

11–20 of 28 posts

Re: Show HN: st – simple statistics from the command line

#11

Post by Doches »

imurray wrote: post #3

For casual purposes st may be convenient, but it doesn't have state of the art numerical stability: my $variance = $count > 1 ? ($sum_square - ($sum**2/$count)) / ($count-1) : undef; Taking the difference between two similar numbers loses precision, and in extreme cases squaring the raw numbers could cause overflow. For comparison, see the recently posted: http://www.python.org/dev/peps/pep-0450/ and https://en.wikip…

If you need numerical stability, I'd use Gary Perlmann's [|stat](http://oldwww.acm.org/perlman/stat/history.html). It's older and somewhat harder to get a copy of, but it's as reliably correct as a piece of software can be...

Re: Show HN: st – simple statistics from the command line

#12

Post by nferraz »

fsiefken wrote: post #9

Nice, is there a maximum rowcount? What would be nice is a way to do a sum on a second or third column - or would you use awk to get those and pipe the result to st?

There isn't a max rowcount for sum, mean, variance, etc, because it is not necessary to hold the data in memory.

The calculation of median and quartiles require that the whole set is stored and later sorted, so it is limited to the available memory.

Regarding your suggestion -- I'm considering the idea of dealing with multiple columns and even CSV and other types of tabulated data.

Re: Show HN: st – simple statistics from the command line

#13

Post by nferraz »

Sprint wrote: post #6
nferraz wrote: post #5

Earlier quoted context omitted.

I like octave and R! The reason I wrote this script was to get quick results from the command line. For instance: I could use grep, cut and other unix tools to get the numbers from a file and make quick calculations. Of course, for complex processing I would use octave or R.

Yeah, I was thinking about that and spent the past minutes to make me some Bash functions like: function mean() { octave -q --eval "mean = mean(load('$1'))" } Then just run "mean numbers.txt". I am sure your approach is much quicker, octave takes a good 0.5s(!) to load on my machine.

Yup, octave requires more time to warm up.

Regarding speed, for simple calculations like sum, mean and variance, the bottleneck is in I/O.

Re: Show HN: st – simple statistics from the command line

#14

Post by nferraz »

asgeirn wrote: post #8
Sprint wrote: post #2

I'd just use octave. It's as simple as $ octave octave:1> a=load('numbers.txt'); octave:2> sum(a) ans = 55 octave:3> mean(a) ans = 5.5000 octave:4> std(a) ans = 3.0277 octave:5> quantile(a) ans = 1.0000 3.0000 5.5000 8.0000 10.0000 etc

Would you be able to use Octave for reading from stdin?

Yes, Sprint suggested this:

    octave -q --eval "mean = mean(load('$1'))"
But, again, octave requires more time to warm up...

Re: Show HN: st – simple statistics from the command line

#16

Post by Kliment »

nferraz wrote: post #7
sprayk wrote: post #4

suckless' terminal emulator already uses the name st, though it's not quite popular enough to be in any major repos. http://st.suckless.org/

Thanks for the information! I wanted to use "stat", but it was already used (display file status); "statistics" was too big. Just as curiosity, I got the idea for this script when I wanted to calculate the sum of some numbers and discovered that the "sum" command was used for another purpose (display file checksums and block counts)!

sta seems to be available

Re: Show HN: st – simple statistics from the command line

#17

Post by nferraz »

riffraff wrote: post #15

maybe you are not aware of it, but there is a nifty little tool in freebsd called ministat that somewhat overlaps with what you did, maybe of interest: http://www.freebsd.org/cgi/man.cgi?query=ministat&apropos=0&...

Nice! They definitely overlap, although there are a few differences... I'm not sure if ministat accepts bignum, scientific notation, etc.

Re: Show HN: st – simple statistics from the command line

#18

Post by imurray »

Doches wrote: post #11
imurray wrote: post #3

For casual purposes st may be convenient, but it doesn't have state of the art numerical stability: my $variance = $count > 1 ? ($sum_square - ($sum**2/$count)) / ($count-1) : undef; Taking the difference between two similar numbers loses precision, and in extreme cases squaring the raw numbers could cause overflow. For comparison, see the recently posted: http://www.python.org/dev/peps/pep-0450/ and https://en.wikip…

If you need numerical stability, I'd use Gary Perlmann's [|stat]( http://oldwww.acm.org/perlman/stat/history.html ). It's older and somewhat harder to get a copy of, but it's as reliably correct as a piece of software can be...

I've just had a look at the |STAT source, and it computes the variance with

        double  M       = Sum/N;                /* mean */
        double  var     = (s2 - M*Sum)/(N-1);   /* variance */
where s2 is the sum of squares. In most reasonable situations, this approach will work fine. It just doesn't take as wide a variety of inputs as is easily possible to achieve with normal floating point doubles. In fairness, the |STAT terms and conditions state:

|STAT PROGRAMS HAVE NOT BEEN VALIDATED FOR LARGE DATASETS, HIGHLY VARIABLE DATA, NOR VERY LARGE NUMBERS.

Re: Show HN: st – simple statistics from the command line

#20

Post by nidoran »

fsiefken wrote: post #9

Nice, is there a maximum rowcount? What would be nice is a way to do a sum on a second or third column - or would you use awk to get those and pipe the result to st?

cut is also a good option when dealing with column-based data. Just specify the delimiter and which column you want.
Post reply on HN