Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PEP 450: Adding A Statistics Module To The Standard Library

python.org

Post reply on HN
Print view View original

51–60 of 85 posts

Re: PEP 450: Adding A Statistics Module To The Standard Library

#51

Post by malkarouri »

bachback wrote: post #34
cdavid wrote: post #28

Earlier quoted context omitted.

numpy is quite portable, I am not sure what you mean by not nearly portable enough. The reason why you can't do pip install numpy is pip's fault, there is nothing that numpy can do to make that work. Note that easy_install numpy does work on windows (without the need for a C compiler).

the problems I have had with numpy are endless. Usually I'll just prefer to write my own, because it's quicker. if you have tried to get numpy running on a cloud machine you'll know what I'm talking about. basically you will have to know how to compile from source, know some gcc, etc. the last time I tried to get it running I promised myself never to use numpy again.

I suggest you try again. I think the situation has changed. I have been using Numpy/Scipy for a few years now, with all the negative experience of having to sort out whether to use g77 or gfortran and the right gcc flags etc. The most recent version, Numpy 1.7.0, is the first one that installed for me smoothly on Windows and OS X (official packages). YMMV, but I think it is alright now.

Re: PEP 450: Adding A Statistics Module To The Standard Library

#52

Post by dandellion »

lutusp wrote: post #24
daniel-levin wrote: post #23

Earlier quoted context omitted.

Agreed. I'm studying statistics at the moment and I'm continually reminded of how easy it is to choose the wrong model / distribution and be incorrect because of some non-obvious and technical reason. For example, just the other day, I wanted to use the binomial distribution to solve a problem. To use this distribution, the trials must be independent of one another. In that particular problem, there was a subtle cond…

> Statistics is way too easy to break to be used naively. Fair enough, but the same argument could be made about using an unskewed standard distribution on non-symmetrical datasets, a common error even among people who should know better. I think binomial functions should be included, on the ground that they're very useful and their probability of misuse is only equal to the continuous statistical forms, not more so.

Hell, sometimes they use a dictionary when they should be using a list. Almost everything can be used wrongly by a begginer, which doesn't mean it shouldn't be there.

I think having a basic stats module always handy would be very convenient.

Re: PEP 450: Adding A Statistics Module To The Standard Library

#53

Post by gknoy »

dandellion wrote: post #52
lutusp wrote: post #24

Earlier quoted context omitted.

> Statistics is way too easy to break to be used naively. Fair enough, but the same argument could be made about using an unskewed standard distribution on non-symmetrical datasets, a common error even among people who should know better. I think binomial functions should be included, on the ground that they're very useful and their probability of misuse is only equal to the continuous statistical forms, not more so.

Hell, sometimes they use a dictionary when they should be using a list. Almost everything can be used wrongly by a begginer, which doesn't mean it shouldn't be there. I think having a basic stats module always handy would be very convenient.

While you are correct, it sounds like his point was that even to someone moderately skilled it is easy to make a mistake that makes your work __completely__ invalid, rather than merely inefficient or too-complicated.

Re: PEP 450: Adding A Statistics Module To The Standard Library

#54

Post by cdavid »

mistercow wrote: post #45
cdavid wrote: post #33

Earlier quoted context omitted.

There is room for improvement, and it gets improved all the time (e.g. openblas is a recent contender). LAPACK is essentially an API for linear algebra, which is what allowed people to improve implementations and to benefit from them in older programs. Think of it as the C library of numerical computing.

I'm confused how a library written in Fortran 90 can be called "the C library of numerical computing".

I was comparing LAPACK to the C library: nobody claims it is weird to use the C library, written > 30 years ago, instead of using something 'more modern'. Everybody uses the C library for system/low level programming, the same is true in numerical computing w.r.t. blas/lapack. Numpy/scipy use it, R use it, julia use it, matlab use it, octave use it.

Re: PEP 450: Adding A Statistics Module To The Standard Library

#55

Post by cdavid »

asgard1024 wrote: post #22
mvanveen wrote: post #13

Earlier quoted context omitted.

numpy has all sorts of awful C bindings which make it less than versatile in environments where you want pure Python. It's great from a performance point of view, but horrible for compatibility. Google App Engine used to suffer because of this (more specifically, it still only restricts your runtime to pure Python, but now you can import numpy at least). I believe the PyPy folks have also had their own set of struggl…

How does that work for SQLite 3, which _is_ part of Python library? I would actually prefer to have numpy included before those statistics functions.

numpy allows you to do some pretty low level stuff (like creating buffer from a raw pointer), which GAE wants to avoid for obvious security issues. Same rationale as why ctypes (parts of the stdlib) is not available.

I have never used numpy on GAE, but I would suspect some of it is not enabled for those reasons.

Re: PEP 450: Adding A Statistics Module To The Standard Library

#57

Post by andrewflnr »

fiatmoney wrote: post #14

It's not a terrible idea to support the absolute basics like mean & variance, but anything beyond that (particularly things like models or tests) is not a good idea for a standard library. Once you hit even something simple like a linear regression you have issues of how to represent missing or discrete variables, handling colinearity, or whether to do online or batch modes which can give different results. Tests in…

  ...how to represent missing or discrete variables...
Don't. Just say no. Just give me the simple easy stuff. Most of us will be fine, and everyone else will know they need something better and won't bother.

Re: PEP 450: Adding A Statistics Module To The Standard Library

#58

Post by lutusp »

dandellion wrote: post #52
lutusp wrote: post #24

Earlier quoted context omitted.

> Statistics is way too easy to break to be used naively. Fair enough, but the same argument could be made about using an unskewed standard distribution on non-symmetrical datasets, a common error even among people who should know better. I think binomial functions should be included, on the ground that they're very useful and their probability of misuse is only equal to the continuous statistical forms, not more so.

Hell, sometimes they use a dictionary when they should be using a list. Almost everything can be used wrongly by a begginer, which doesn't mean it shouldn't be there. I think having a basic stats module always handy would be very convenient.

> I think having a basic stats module always handy would be very convenient.

I absolutely agree. My only point was that these tools are sometimes misapplied, not at all to argue that they shouldn't be readily available. They should be.

Re: PEP 450: Adding A Statistics Module To The Standard Library

#59

Post by alexholehouse »

ot wrote: post #21

Just out of curiosity, I submitted this yesterday: https://news.ycombinator.com/item?id=6190603 The URL was http://www.python.org/dev/peps/pep-0450/ While this is http://www.python.org/dev/peps/pep-0450 That is, exactly the same except for a trailing slash. Doesn't the deduplication algorithm handle this case?

I posted an ASK PG [1] about this last month ago and got an angry email from (presumably) a mod. His point was totally valid - that PG 'aint go time fo' that' which is true, but I was just hoping to draw attention to it, as opposed to demand PG drop what he's doing right now and fix it. To be honest, I was quite surprised by the tone of the email.

[1] https://news.ycombinator.com/item?id=5908075

Re: PEP 450: Adding A Statistics Module To The Standard Library

#60

Post by lutusp »

enalicho wrote: post #44
lutusp wrote: post #18

Great idea, but while assembling this library, don't leave out permutations, combinations, and the binomial Probability Mass Function (PMF) and Cumulative Distribution Function (CDF). Small overhead, easy to implement, very useful. More here: http://arachnoid.com/binomial_probability

Permutations and combinations already exist within the itertools module.

> Permutations and combinations already exist within the itertools module.

Not exactly. Given argument lists, Itertools provides result lists (actually, iterators for that purpose) with the original elements permuted and combined, but doesn't provide numerical results for numerical arguments, as shown here: http://arachnoid.com/binomial_probability

I was referring to permutation and combination mathematical functions, not generator functions.

Post reply on HN