Live data from Hacker News

Thoughts on porting NumPy to PyPy

technicaldiscovery.blogspot.com

11–20 of 34 posts

Re: Thoughts on porting NumPy to PyPy

#11
post #6

I just don't see adding a special matrix infix operator to Python happening. It is too specialized, and yet matrix is not special enough that it must have its own operator. Isn't this what operator overloading is for? I suspect the real problem is NumPy's type system implementation, since data-types are not visible as different Python types.

[deleted]

Re: Thoughts on porting NumPy to PyPy

#12
If numpy-c were to be called over ctypes, it'd be slow.The reason PyPy's numpy reimplementation is so insanely fast is the integration with the JIT.

A more plausible scenario would be both CPython and PyPy using numpy-py to call into numpy-c and numpy-pypy respectively.

Re: Thoughts on porting NumPy to PyPy

#13
post #3

I couldn't agree more. This is related to another recent HN posting ( http://news.ycombinator.com/item?id=3104598 ) and discussions at PyCodeConf in Miami. There's a general feeling (which I've experienced in many of my interactions with non-scientific Python programmers) that the folks working on pure Python don't really "get" the scientific Python community. We would all understand each other better if the PyPy fol…

Note that as PyPy we deliberately don't take part in the language design discussions. We won't implement the array infix operator precisely because other people make those decisions.

Re: Thoughts on porting NumPy to PyPy

#14
Off-topic but anyone else seeing every instance of "fi" in the text replaced with a slightly overlapping AV? (I tried to cut and paste it but it pasted as just "fi").

I'm on Ubuntu Ocelot & Firefox 7.

nevermind: I'm seeing it on various websites that use ligatures. Possibly because I'm half-way through an Ubuntu upgrade.

Re: Thoughts on porting NumPy to PyPy

#15
post #3

I couldn't agree more. This is related to another recent HN posting ( http://news.ycombinator.com/item?id=3104598 ) and discussions at PyCodeConf in Miami. There's a general feeling (which I've experienced in many of my interactions with non-scientific Python programmers) that the folks working on pure Python don't really "get" the scientific Python community. We would all understand each other better if the PyPy fol…

I spoke with Travis on Friday at Enthought and he opened my eyes to some of the possible problems of the pypy-numpy effort (which I've started discussing: http://mail.python.org/pipermail/pypy-dev/2011-October/00860... ). I hadn't realised that the port might exclude use of the rest of SciPy, this strikes me as a massive missed opportunity if it comes to pass. Perhaps a few other knowledgeable folk could post in the pypy-dev thread so we can reasonably understand the limitations and possibilities of the various approaches?

Re: Thoughts on porting NumPy to PyPy

#16
post #2

I love it when someone writes an articulate, non-inflamatory blog post with reasonable suggestions on how to improve things. It's sad how rare this is around here these days.

I missed the specific suggestions on how the pypy guys could improve things - other than "do a lot of work on numpy itself as well so that it better supports multiple implementations" (paraphrase obviously). That really needs to be the side of the equation that the numpy team take hold of.

Re: Thoughts on porting NumPy to PyPy

#17
post #3

I couldn't agree more. This is related to another recent HN posting ( http://news.ycombinator.com/item?id=3104598 ) and discussions at PyCodeConf in Miami. There's a general feeling (which I've experienced in many of my interactions with non-scientific Python programmers) that the folks working on pure Python don't really "get" the scientific Python community. We would all understand each other better if the PyPy fol…

I spoke with Travis on Friday at Enthought and he opened my eyes to some of the possible problems of the pypy-numpy effort (which I've started discussing: http://mail.python.org/pipermail/pypy-dev/2011-October/00860... ). I hadn't realised that the port might exclude use of the rest of SciPy, this strikes me as a massive missed opportunity if it comes to pass. Perhaps a few other knowledgeable folk could post in the…

I don't really know what you mean by supporting numpy on pypy "excludes" scipy (other than in the short term). If you mean you assumed that the pypy team would port all of scipy as well as numpy, that seems like an unrealistic expectation for an initial port!

An initial port seems like the only way forward from the point of view of the pypy team. I think it is unrealistic to expect the pypy team to take on the work of changing numpy so that it is more friendly to alternative implementations. I would certainly expect them to be involved in the discussion though.

The article also seems to miss that there is work ongoing to bring pypy support to Cython.

Re: Thoughts on porting NumPy to PyPy

#18
post #8
post #6

I just don't see adding a special matrix infix operator to Python happening. It is too specialized, and yet matrix is not special enough that it must have its own operator. Isn't this what operator overloading is for? I suspect the real problem is NumPy's type system implementation, since data-types are not visible as different Python types.

If you spent a day doing some serious linear algebra in Python you might change your tune. * (multiplication) between NumPy arrays by default does element-wise multiplication (potentially with broadcasting), which is the desired default behavior. In R, for example, you can define custom infix operators so that a * b is elementwise but a %* % b is matrix multiplication. If you write down a complicated linear algebra e…

I see your point, and it is why I tend to like operator overloading in languages.

However, Python is a general purpose language, and scientific computing is a single domain. It is used in many different domains. Consider that there are many changes that individual communities would like, and if Python granted all of those requests, the language would be a mess. That's the challenge in designing a general purpose programming language.

Re: Thoughts on porting NumPy to PyPy

#19

Earlier quoted context omitted.

I spoke with Travis on Friday at Enthought and he opened my eyes to some of the possible problems of the pypy-numpy effort (which I've started discussing: http://mail.python.org/pipermail/pypy-dev/2011-October/00860... ). I hadn't realised that the port might exclude use of the rest of SciPy, this strikes me as a massive missed opportunity if it comes to pass. Perhaps a few other knowledgeable folk could post in the…

I don't really know what you mean by supporting numpy on pypy "excludes" scipy (other than in the short term). If you mean you assumed that the pypy team would port all of scipy as well as numpy, that seems like an unrealistic expectation for an initial port! An initial port seems like the only way forward from the point of view of the pypy team. I think it is unrealistic to expect the pypy team to take on the work o…

Well, the big problem to solve with SciPy is the fact that there is more, respectively, of C, C++, and Fortran in the SciPy codebase than Python (http://www.ohloh.net/p/scipy). Part of why Python has succeeded in scientific computing is integration of legacy codebases (via f2py or C extensions or...). There are probably man-years of work involved in devising a solution-- which by the time it's complete may be basically irrelevant or, worse, fragment the community. I personally think that going down this rabbit hole (porting 10 years of scientific Python libraries to PyPy) would amount to an exercise in vanity rather than producing the kinds of revolutionary changes to array-oriented computing that need to happen soon to deal with the large-scale data processing challenges of the present and future. Having recently used GPUs to speed up statistical inference algorithms by a factor of 50 or more, I am not that motivated by a JIT beating C in some cases (as Travis wrote: "C speed is the wrong target"). Many in the SciPy community are convinced that NumPy will not provide the computational foundation that we need going forward, and they are going to step up and start building the next generation NumPy (or whatever it's going to be called). We'd rather have more of the smartest computer scientists in the Python community focused on this problem (building more sophisticated data processing pipelines for use in Python) than on speeding up the Python code that by my estimation doesn't matter that much.

Re: Thoughts on porting NumPy to PyPy

#20
post #19

Earlier quoted context omitted.

I don't really know what you mean by supporting numpy on pypy "excludes" scipy (other than in the short term). If you mean you assumed that the pypy team would port all of scipy as well as numpy, that seems like an unrealistic expectation for an initial port! An initial port seems like the only way forward from the point of view of the pypy team. I think it is unrealistic to expect the pypy team to take on the work o…

Well, the big problem to solve with SciPy is the fact that there is more, respectively, of C, C++, and Fortran in the SciPy codebase than Python ( http://www.ohloh.net/p/scipy ). Part of why Python has succeeded in scientific computing is integration of legacy codebases (via f2py or C extensions or...). There are probably man-years of work involved in devising a solution-- which by the time it's complete may be basic…

AFAIUI, what you're missing is that PyPy can (and often does) interface directly to C libraries, from RPython. So the prospect of re-implementing those specialized codebases isn't a real issue: only the CPython-API based wrappers would need re-implementing. I believe those are a small part of the total code you mention.
Post reply on HN