Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PyPy funded to begin support for Python 3 and Numpy

morepypy.blogspot.com

Post reply on HN
Print view View original

21–30 of 32 posts

Re: PyPy funded to begin support for Python 3 and Numpy

#21

Post by kingkilr »

MichaelSalib wrote: post #18
fijal wrote: post #14

Earlier quoted context omitted.

I think I disagree pretty much about every single point you make. First for something as simple as laplace equation solver numpy vectorized loop is 35ms per loop vs 6.3ms for C. As your list of operations increase, your need of intermediates grow and your speed decreases, but let's not go to details. Obviously if you just call a vendor-optimized library, you can use whatever you feel like and it'll be equally good, b…

First for something as simple as laplace equation solver numpy vectorized loop is 35ms per loop vs 6.3ms for C. Isn't numexpr a good solution for that problem? numexpr is much much simpler than PyPy, so if it can reduce the performance overhead of complex vector operations with loop fusion, that seems a huge win.

[citation needed] as for numexpr being simpler than PyPy. Using the dumbest measure of simplicity, total code size, it fails: PyPy's NumPy is 167KB of code, NumExpr is 181KB.

Re: PyPy funded to begin support for Python 3 and Numpy

#22

Post by kingkilr »

pwang wrote: post #20
fijal wrote: post #17

Earlier quoted context omitted.

Note: this reply is on the wrong level, I cannot reply correctly. I don't think I'm highly dismissive about numpy/scipy/cython community. I wouldn't be implementing all this stuff if I didn't think those APIs are good and they're the future of scientific computing, they just lack a reasonable replacement for C. If Python is to surpress Fortran on the scientific field, it really does need a way to express fast algorit…

"...they just lack a reasonable replacement for C" Actually, Fijal, I have a question - and this is related to our previous Skype discussion as well. I know that a lot of the work on PyPy has been on the JIT, but have you guys really ever pursued the idea of just building PyPy as a front end for LLVM? All your type inferencing logic would probably be a lot more code than what a traditional LLVM front-end normally con…

LLVM has been looked at, as both a backened for RPython as well as for the JIT, I've written down some of the reasons it's not appropriate for PyPy (particularly WRT the JIT) here: http://www.quora.com/LLVM/Is-LLVM-not-good-for-interpreted-l...

Re: PyPy funded to begin support for Python 3 and Numpy

#23

Post by MichaelSalib »

kingkilr wrote: post #21
MichaelSalib wrote: post #18

Earlier quoted context omitted.

First for something as simple as laplace equation solver numpy vectorized loop is 35ms per loop vs 6.3ms for C. Isn't numexpr a good solution for that problem? numexpr is much much simpler than PyPy, so if it can reduce the performance overhead of complex vector operations with loop fusion, that seems a huge win.

[citation needed] as for numexpr being simpler than PyPy. Using the dumbest measure of simplicity, total code size, it fails: PyPy's NumPy is 167KB of code, NumExpr is 181KB.

numexpr is much simpler than pypy. I never said that it was much simpler than pypy's incomplete numpy support.

Re: PyPy funded to begin support for Python 3 and Numpy

#24

Post by pwang »

kingkilr wrote: post #21
MichaelSalib wrote: post #18

Earlier quoted context omitted.

First for something as simple as laplace equation solver numpy vectorized loop is 35ms per loop vs 6.3ms for C. Isn't numexpr a good solution for that problem? numexpr is much much simpler than PyPy, so if it can reduce the performance overhead of complex vector operations with loop fusion, that seems a huge win.

[citation needed] as for numexpr being simpler than PyPy. Using the dumbest measure of simplicity, total code size, it fails: PyPy's NumPy is 167KB of code, NumExpr is 181KB.

Um, that's not a dumb measure of simplicity, it's just a measure of code size. You might as well look at the average number of characters in the names of the authors of the two projects; it would be equally irrelevant.

Here is a real measure of simplicity: how long does it take to explain to a Numpy user how to wrap an array expression in a string, versus explaining how a JITting compiler compiler works and how to interface its runtime to their existing Python installation and how to build it and what the limitations of RPython are.

Heck, I'm an actual developer (not a scientific programmer) and it took me a little while to understand what PyPy does.

Re: PyPy funded to begin support for Python 3 and Numpy

#25

Post by pwang »

kingkilr wrote: post #22
pwang wrote: post #20

Earlier quoted context omitted.

"...they just lack a reasonable replacement for C" Actually, Fijal, I have a question - and this is related to our previous Skype discussion as well. I know that a lot of the work on PyPy has been on the JIT, but have you guys really ever pursued the idea of just building PyPy as a front end for LLVM? All your type inferencing logic would probably be a lot more code than what a traditional LLVM front-end normally con…

LLVM has been looked at, as both a backened for RPython as well as for the JIT, I've written down some of the reasons it's not appropriate for PyPy (particularly WRT the JIT) here: http://www.quora.com/LLVM/Is-LLVM-not-good-for-interpreted-l...

Thanks for the link!

Re: PyPy funded to begin support for Python 3 and Numpy

#26

Post by tylerritchie »

pwang wrote: post #24
kingkilr wrote: post #21

Earlier quoted context omitted.

[citation needed] as for numexpr being simpler than PyPy. Using the dumbest measure of simplicity, total code size, it fails: PyPy's NumPy is 167KB of code, NumExpr is 181KB.

Um, that's not a dumb measure of simplicity, it's just a measure of code size. You might as well look at the average number of characters in the names of the authors of the two projects; it would be equally irrelevant. Here is a real measure of simplicity: how long does it take to explain to a Numpy user how to wrap an array expression in a string, versus explaining how a JITting compiler compiler works and how to in…

>I'm an actual developer (not a scientific programmer) and it took me a little while to understand what PyPy does.

I'm a scientist, not a scientific programmer or a developer and this is all I really care about: PyPy is currently--in it's partially implemented state--much, much faster than CPython on the vast majority of things it can do. If I am able to use PyPy's NumPy and it's faster than traditional NumPy I will do so as long as the opportunity cost doesn't outweigh the speed increases (NumPy is pretty useless to me--maybe not some--without SciPy and matplotlib).

I don't care that PyPy is written in RPython any more than I care that it has a JIT, or that CPython is written in C. I also don't care how that JIT works or how CPython compiles to byte code or how Jython does magic to make my Python code run on the JVM. I do care that it "works," as a scientist. I do care that they are "correct" implementations, as a scientist. As an individual, I am interested in the inner workings of PyPy and CPython, CoffeeScript, Go, and Brain Fuck, but when I'm working on research the only thing that actually is important as far as the language implementation is concerned is that it just works. The interpreter is just a brand of the particular tool that I'm using.

I would certainly prefer it if PyPy was 100% compatible with the CPython C API even if it was at 80% (maybe even 60%) of the CPython C API speed because then I don't even have to think. I'd be using PyPy because it's faster overall and I can do the analyses I want faster.

Anyway, I think if you're explaining all of what you mentioned to a NumPy user or a PyPy NumPy user you'd be doing it wrong. Or maybe the PyPy folks would be doing it wrong. Because this is how that conversation would go with my peers.

  Sad Panda: "Ugh my code is running slowly I think I have to jump into C"
  Me: "Have you tried PyPy's NumPy yet?"
  Sad Panda: "What's that?"
  Me: "It's faster Python and NumPy. Go here [link] and download it see if it runs your code faster"
  Sad Panda: "Okay I'll do that"
  ..a while later..
  Sad Panda: "It was a little faster, but I ended up getting one of the CS guys to help me run it on a tesla cluster with OpenCL. But I think I can use it on the spiny lumpsucker data I'm collecting."

Re: PyPy funded to begin support for Python 3 and Numpy

#27

Post by fijal »

pwang wrote: post #24
kingkilr wrote: post #21

Earlier quoted context omitted.

[citation needed] as for numexpr being simpler than PyPy. Using the dumbest measure of simplicity, total code size, it fails: PyPy's NumPy is 167KB of code, NumExpr is 181KB.

Um, that's not a dumb measure of simplicity, it's just a measure of code size. You might as well look at the average number of characters in the names of the authors of the two projects; it would be equally irrelevant. Here is a real measure of simplicity: how long does it take to explain to a Numpy user how to wrap an array expression in a string, versus explaining how a JITting compiler compiler works and how to in…

why the people wrapping stuff in strings have to understand the limitations of RPython? It's "wrap expression in strings" vs "do nothing".

Re: PyPy funded to begin support for Python 3 and Numpy

#28

Post by lmm »

tylerritchie wrote: post #26
pwang wrote: post #24

Earlier quoted context omitted.

Um, that's not a dumb measure of simplicity, it's just a measure of code size. You might as well look at the average number of characters in the names of the authors of the two projects; it would be equally irrelevant. Here is a real measure of simplicity: how long does it take to explain to a Numpy user how to wrap an array expression in a string, versus explaining how a JITting compiler compiler works and how to in…

>I'm an actual developer (not a scientific programmer) and it took me a little while to understand what PyPy does. I'm a scientist, not a scientific programmer or a developer and this is all I really care about: PyPy is currently--in it's partially implemented state--much, much faster than CPython on the vast majority of things it can do. If I am able to use PyPy's NumPy and it's faster than traditional NumPy I will…

>I would certainly prefer it if PyPy was 100% compatible with the CPython C API even if it was at 80% (maybe even 60%) of the CPython C API speed because then I don't even have to think. I'd be using PyPy because it's faster overall and I can do the analyses I want faster.

While part of me agrees with this, if PyPy starts sacrificing performance for CPython compatibility then pretty soon it'll degenerate into CPython.

Re: PyPy funded to begin support for Python 3 and Numpy

#29

Post by [deleted] »

ColdAsIce wrote: post #2

I like pypy and its ambitions, last time I tested it, about a month ago it was very speedy and startup time considerable faster than Cpython. However the regex exercises I wanted to do couldnt be done. Pypy seemingly didnt have a good regexp engine. If I remember correctly, something with groups and backwards-reference... Has that changed?

[deleted]

Re: PyPy funded to begin support for Python 3 and Numpy

#30

Post by andrewcooke »

ColdAsIce wrote: post #2

I like pypy and its ambitions, last time I tested it, about a month ago it was very speedy and startup time considerable faster than Cpython. However the regex exercises I wanted to do couldnt be done. Pypy seemingly didnt have a good regexp engine. If I remember correctly, something with groups and backwards-reference... Has that changed?

i have a pure python regexp implementation that you can use in pypy (and it passes almost every test used by the cpython re package - only the LOCALE flag is not supported). however, it's around 100x[1] slower that the re package (on simple matches; it's not backtracking so it can be similar in speed for pathological cases).

also, when i looked at pypy the re implementation was in c and looked so similar to cpython's i assumed it was the same or closely related. but i didn't pay much attention, so i may be mistaken.

[1] roughly. it actually runs 6x faster on recent pypy that cpython.

Post reply on HN