Reminds me of the story that made rounds here couple of years ago: The Python Standard Library - Where Modules Go To Die
PEP 450: Adding A Statistics Module To The Standard Library
11–20 of 85 posts
Re: PEP 450: Adding A Statistics Module To The Standard Library
#12Nice proposal. I think the problem is numpy itself. If you could just do pip install numeric_package then nobody can complain. I don't quite understand why a package has to depend on LINPACK. I will probably switch to julia-lang, because numpy is (at least for me) not that great to work with.
NumPy is a full MAT-LAB for Python, not a simple drop-in statistics library. It has to depend on LINPACK because writing a full linear algebra library that performs well is damn hard and takes several researcher-years. Most serious scientific computing libraries and utilities depend on it, including Julia I think. There is certainly room for simpler libraries for people not seriously into numeric computations, as the document linked well indicates.
Re: PEP 450: Adding A Statistics Module To The Standard Library
#13Nice proposal. I think the problem is numpy itself. If you could just do pip install numeric_package then nobody can complain. I don't quite understand why a package has to depend on LINPACK. I will probably switch to julia-lang, because numpy is (at least for me) not that great to work with.
numpy has all sorts of awful C bindings which make it less than versatile in environments where you want pure Python. It's great from a performance point of view, but horrible for compatibility.
Google App Engine used to suffer because of this (more specifically, it still only restricts your runtime to pure Python, but now you can import numpy at least). I believe the PyPy folks have also had their own set of struggles with numpy compatibility, although I'm not sure what the state of that is at present.
In any case, I think these compatibility concerns alone make a strong argument for including simple Statistics tooling into the standard library.
Re: PEP 450: Adding A Statistics Module To The Standard Library
#14It's not a terrible idea to support the absolute basics like mean & variance, but anything beyond that (particularly things like models or tests) is not a good idea for a standard library. Once you hit even something simple like a linear regression you have issues of how to represent missing or discrete variables, handling colinearity, or whether to do online or batch modes which can give different results. Tests in particular are fraught because if you're going to make them available for general consumption they need a good explanation of when they're appropriate, which is basically a semester course in statistics and well out of scope for standard library docs.
Basically, the idea of "batteries included" should also mean that if something looks like you can put a D-cell in there, you're unlikely to blow your arm off.
Re: PEP 450: Adding A Statistics Module To The Standard Library
#15> For many people, installing numpy may be difficult or impossible. For example, people in corporate environments may have to go through a difficult, time-consuming process before being permitted to install third-party software.
I do not regard this as a good justification for putting something in the standard library! If you don't have root access, use vitualenv (which you might want to do anyway) and install the package somewhere under your home directory.
Re: PEP 450: Adding A Statistics Module To The Standard Library
#16About damned time. Writing your own stats library is like writing your own crypto.
You wouldn't write your own - numpy / scipy have everything you'll need.
Re: PEP 450: Adding A Statistics Module To The Standard Library
#17> For many people, installing numpy may be difficult or impossible. For example, people in corporate environments may have to go through a difficult, time-consuming process before being permitted to install third-party software. I do not regard this as a good justification for putting something in the standard library! If you don't have root access, use vitualenv (which you might want to do anyway) and install the pa…
Having the technical capability to do something is not the same as having permission to do it. The PEP refers to corporate policies. Installing third-party software in your home directory would be just as much a violation as doing it in a location that requires root.
Re: PEP 450: Adding A Statistics Module To The Standard Library
#18Great idea, but while assembling this library, don't leave out permutations, combinations, and the binomial Probability Mass Function (PMF) and Cumulative Distribution Function (CDF). Small overhead, easy to implement, very useful. More here:
Re: PEP 450: Adding A Statistics Module To The Standard Library
#19> For many people, installing numpy may be difficult or impossible. For example, people in corporate environments may have to go through a difficult, time-consuming process before being permitted to install third-party software. I do not regard this as a good justification for putting something in the standard library! If you don't have root access, use vitualenv (which you might want to do anyway) and install the pa…
NumPy is not nearly portable enough to do what you are describing as a user on a windows machine. You cannot simply do a `pip install numpy` into a virtualenv. Instead you must either install the package system-wide, or compile it yourself, which means getting a working MinGW environment or similar.
Edit: Although, I do agree that NumPy being difficult to install is not, on its own, a good justification for the PEP.
Re: PEP 450: Adding A Statistics Module To The Standard Library
#20Batteries included is a fine philosophy when starting a language to encourage early adoption, but at this point, I don't think it's worth adding new libraries to the stdlib. Here's why:
- It's very easy to find and install third party modules
- Once a library is added to stdlib, the API is essentially frozen. This means we can end up stuck with less than ideal APIs (shutil/os, urllib2/urrlib, etc) or Guido & co are stuck in a time consuming PEP/deprecate/delete loop for even minor API improvements.
- libraries outside of the stdlib are free to evolve. users of those libraries who don't want to stay on the bleeding edge are free to stay on old versions.