Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PEP 450: Adding A Statistics Module To The Standard Library

python.org

Post reply on HN
Print view View original

21–30 of 85 posts

Re: PEP 450: Adding A Statistics Module To The Standard Library

#21

Post by ot »

Just out of curiosity, I submitted this yesterday:

https://news.ycombinator.com/item?id=6190603

The URL was

    http://www.python.org/dev/peps/pep-0450/
While this is

    http://www.python.org/dev/peps/pep-0450
That is, exactly the same except for a trailing slash. Doesn't the deduplication algorithm handle this case?

Re: PEP 450: Adding A Statistics Module To The Standard Library

#22

Post by asgard1024 »

mvanveen wrote: post #13
bachback wrote: post #6

Nice proposal. I think the problem is numpy itself. If you could just do pip install numeric_package then nobody can complain. I don't quite understand why a package has to depend on LINPACK. I will probably switch to julia-lang, because numpy is (at least for me) not that great to work with.

numpy has all sorts of awful C bindings which make it less than versatile in environments where you want pure Python. It's great from a performance point of view, but horrible for compatibility. Google App Engine used to suffer because of this (more specifically, it still only restricts your runtime to pure Python, but now you can import numpy at least). I believe the PyPy folks have also had their own set of struggl…

How does that work for SQLite 3, which _is_ part of Python library?

I would actually prefer to have numpy included before those statistics functions.

Re: PEP 450: Adding A Statistics Module To The Standard Library

#23

Post by daniel-levin »

fiatmoney wrote: post #14

It's not a terrible idea to support the absolute basics like mean & variance, but anything beyond that (particularly things like models or tests) is not a good idea for a standard library. Once you hit even something simple like a linear regression you have issues of how to represent missing or discrete variables, handling colinearity, or whether to do online or batch modes which can give different results. Tests in…

Agreed. I'm studying statistics at the moment and I'm continually reminded of how easy it is to choose the wrong model / distribution and be incorrect because of some non-obvious and technical reason. For example, just the other day, I wanted to use the binomial distribution to solve a problem. To use this distribution, the trials must be independent of one another. In that particular problem, there was a subtle condition that made the trials non-independent. I arrived at correct-appearing answers (0 <= P <= 1) that were actually all wrong. Statistics is way too easy to break to be used naively.

Re: PEP 450: Adding A Statistics Module To The Standard Library

#24

Post by lutusp »

daniel-levin wrote: post #23
fiatmoney wrote: post #14

It's not a terrible idea to support the absolute basics like mean & variance, but anything beyond that (particularly things like models or tests) is not a good idea for a standard library. Once you hit even something simple like a linear regression you have issues of how to represent missing or discrete variables, handling colinearity, or whether to do online or batch modes which can give different results. Tests in…

Agreed. I'm studying statistics at the moment and I'm continually reminded of how easy it is to choose the wrong model / distribution and be incorrect because of some non-obvious and technical reason. For example, just the other day, I wanted to use the binomial distribution to solve a problem. To use this distribution, the trials must be independent of one another. In that particular problem, there was a subtle cond…

> Statistics is way too easy to break to be used naively.

Fair enough, but the same argument could be made about using an unskewed standard distribution on non-symmetrical datasets, a common error even among people who should know better.

I think binomial functions should be included, on the ground that they're very useful and their probability of misuse is only equal to the continuous statistical forms, not more so.

Re: PEP 450: Adding A Statistics Module To The Standard Library

#25

Post by dalke »

fiatmoney wrote: post #14

It's not a terrible idea to support the absolute basics like mean & variance, but anything beyond that (particularly things like models or tests) is not a good idea for a standard library. Once you hit even something simple like a linear regression you have issues of how to represent missing or discrete variables, handling colinearity, or whether to do online or batch modes which can give different results. Tests in…

The PEP suggests that the functionality would be comparable to that in a high school calculator or in Excel/LibreOffice/Gnumeric. The existence of that functionality suggests to me that a stats package can be useful even if it doesn't handle things like missing data.

Similarly, Excel/etc. support these functions without a "semester course in statistics." Instead, you'll find that there are many web pages from semester courses in statistics which end up teaching how to use Excel. The same would no doubt happen with Python.

I don't why a statistics standard library module needs to provide a "good explanation of when they're appropriate" to a higher standard than any other module. Python provides trigonometric and hyperbolic functions without teaching trigonometry. It provides complex numbers and cmath without teaching people about complex numbers. It provides several different random distribution functions without teaching anything about Pareto, Weibull, or von Mises distributions.

For that matter, data structures is a semester course as well, but the Python documentation doesn't teach those differences in its documentation of deque, stack, hash table, etc., nor describe algorithms like heapq and bisect.

"whether to do online or batch modes which can give different results". The PEP says it will prefer batch mode:

      Concentrate on data in sequences, allowing two-passes over the data,
      rather than potentially compromise on accuracy for the sake of a one-pass
      algorithm

Re: PEP 450: Adding A Statistics Module To The Standard Library

#26

Post by dalke »

clutchski wrote: post #20

Batteries included is a fine philosophy when starting a language to encourage early adoption, but at this point, I don't think it's worth adding new libraries to the stdlib. Here's why: - It's very easy to find and install third party modules - Once a library is added to stdlib, the API is essentially frozen. This means we can end up stuck with less than ideal APIs (shutil/os, urllib2/urrlib, etc) or Guido & co are s…

Those are good reasons for rejecting any addition to the standard library. However, new libraries are sometimes added to the standard library, which means the reasons you listed can be overcome by even better reasons for inclusion.

What are those reasons for why a new library can be included, and why aren't those reasons appropriate justification for including this proposed statistics package?

Re: PEP 450: Adding A Statistics Module To The Standard Library

#27

Post by Peaker »

dalke wrote: post #26
clutchski wrote: post #20

Batteries included is a fine philosophy when starting a language to encourage early adoption, but at this point, I don't think it's worth adding new libraries to the stdlib. Here's why: - It's very easy to find and install third party modules - Once a library is added to stdlib, the API is essentially frozen. This means we can end up stuck with less than ideal APIs (shutil/os, urllib2/urrlib, etc) or Guido & co are s…

Those are good reasons for rejecting any addition to the standard library. However, new libraries are sometimes added to the standard library, which means the reasons you listed can be overcome by even better reasons for inclusion. What are those reasons for why a new library can be included, and why aren't those reasons appropriate justification for including this proposed statistics package?

> However, new libraries are sometimes added to the standard library, which means the reasons you listed can be overcome by even better reasons for inclusion.

Or overcome by bad judgement.

Re: PEP 450: Adding A Statistics Module To The Standard Library

#28

Post by cdavid »

siddboots wrote: post #19
cabalamat wrote: post #15

> For many people, installing numpy may be difficult or impossible. For example, people in corporate environments may have to go through a difficult, time-consuming process before being permitted to install third-party software. I do not regard this as a good justification for putting something in the standard library! If you don't have root access, use vitualenv (which you might want to do anyway) and install the pa…

NumPy is not nearly portable enough to do what you are describing as a user on a windows machine. You cannot simply do a `pip install numpy` into a virtualenv. Instead you must either install the package system-wide, or compile it yourself, which means getting a working MinGW environment or similar. Edit: Although, I do agree that NumPy being difficult to install is not, on its own, a good justification for the PEP.

numpy is quite portable, I am not sure what you mean by not nearly portable enough.

The reason why you can't do pip install numpy is pip's fault, there is nothing that numpy can do to make that work. Note that easy_install numpy does work on windows (without the need for a C compiler).

Re: PEP 450: Adding A Statistics Module To The Standard Library

#29

Post by cdavid »

bachback wrote: post #6

Nice proposal. I think the problem is numpy itself. If you could just do pip install numeric_package then nobody can complain. I don't quite understand why a package has to depend on LINPACK. I will probably switch to julia-lang, because numpy is (at least for me) not that great to work with.

Numpy does not depend on LINPACK, only scipy does (numpy only uses blas if you have one installed, it is optional).

The reason why scipy (and Julia BTW) need blas/lapack is because that's the only way to have decent performance and reasonably accurate linear algebra. The alternative is writing your own implementation of something that has been used and debugged for 30 years, which does not seem like a good idea.

Re: PEP 450: Adding A Statistics Module To The Standard Library

#30

Post by daGrevis »

ot wrote: post #21

Just out of curiosity, I submitted this yesterday: https://news.ycombinator.com/item?id=6190603 The URL was http://www.python.org/dev/peps/pep-0450/ While this is http://www.python.org/dev/peps/pep-0450 That is, exactly the same except for a trailing slash. Doesn't the deduplication algorithm handle this case?

Technically speaking, they are separate URLs that may lead to separate resources. For example, Google engine treats them as separate URLs.

That's the reason why opening http://www.python.org/dev/peps/pep-0450 redirects to http://www.python.org/dev/peps/pep-0450/ . HN engine should follow redirect to avoid situations like this.

Post reply on HN