Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is anyone using PyPy for real work?

news.ycombinator.com

Post reply on HN
Print view View original

161–170 of 185 posts

Re: Ask HN: Is anyone using PyPy for real work?

#161

Post by ggm »

CyberDildonics wrote: post #154
ggm wrote: post #12

I'm using pypy to analyse 350m DNS events a day, through python cached dicts to avoid dns lookup stalls. I am getting 95% dict cache hit rate, and use threads with queue locks. Moving to pypy definitely speeded me up a bit. Not as much as I'd hoped, it's probably all about string index into dict and dict management. I may recode into a radix tree. Hard to work out in advance how different it would be: People optimise…

Uplift from normal python was trivial. By definition if you lift something it is going to go up, but what does this mean?

If you replace your python engine you have to replace your imports.

Some engines can't build and deploy all imports.

Some engines demand syntactic sugar to do their work. Pypy doesn't

Re: Ask HN: Is anyone using PyPy for real work?

#163

Post by mattip »

ggm wrote: post #159
mattip wrote: post #22

Earlier quoted context omitted.

> it's probably all about string index into dict and dict management Cool. Is the performance here something you would like to pursue? If so could you open an issue [0] with some kind of reproducer? [0] https://foss.heptapod.net/pypy/pypy/-/issues

I'm thinking about how to demonstrate the problem. I have a large pickle but pickle load/dump times across gc.disable()/gc.enable() really doesn't say much. I need to find out how to instrument the seek/add cost of threads against the shared dict under a lock. My gut feel is that probably if I inlined things instead of calling out to functions I'd shave a bit more too. So saying "slower than expected" may be unfair b…

Would love to hear more. You can reach us with any of these methods https://www.pypy.org/contact.html

Re: Ask HN: Is anyone using PyPy for real work?

#164

Post by fragebogen »

I'm running a constrained convex optimization project at work, where we need as close to real time (Basically I'm using a SciPy exclusively for the optimization routine:

* minimize(method="SLSQP") [0]

* A list comprehention which calls ~10-500 pre-fitted PchipInterpolator [1] functions and stores the values as a np.array().

The Pchip functions (and it's first derivatives) are used in the main opt function as well as in several constraints.

Most jobs took about 10 seconds but the long tail might take up to 10 min some times. I tried the pypy 3.8 (7.3.9), and saw similar compute times on the shorter jobs, but roughly ~2x slower compute times on the heavier jobs. This obviously was not what I expected, but I had very limited experience with pypy and didn't know how to debug further.

Eventually python 3.10 came around and gave 1.25x speed increase, and then 3.11 which gave another 1.6-1.7x increase which gave a decent ~2x cumulative speedup, but the occasional heavy jobs still stay in the 5 min range and would have been nicer in the 10-30s obviously.

Still I would like to say that trying pypy out was a quite smooth experience, staying within scipy land, took me half a day to switch and benchmark. But if anyone else has experience with pypy and scipy, knowing some obvious pitfalls, it would be much appreciated to hear.

[0] https://docs.scipy.org/doc/scipy/reference/optimize.minimize...

[1] https://docs.scipy.org/doc/scipy/reference/generated/scipy.i...

Re: Ask HN: Is anyone using PyPy for real work?

#165

Post by oxmane »

mattip wrote: post #117
oxmane wrote: post #115

At Alooma ( https://www.linkedin.com/mwlite/company/alooma ) we've been running all our integrations with data sources using PyPy. Main motivation was indeed performance gains. FWIW, since I've seen it mentioned, we've also been using psycopg2cffi to access Postgres sources. The product now lives (at least partially) as Datastream on GCP ( https://cloud.google.com/datastream/docs/overview ). I'm not sure though if it…

Cool. Yes, I am interested in hearing more.

Sent you an intro email to the relevant person.

Re: Ask HN: Is anyone using PyPy for real work?

#166

Post by mattip »

fragebogen wrote: post #164

I'm running a constrained convex optimization project at work, where we need as close to real time ( Basically I'm using a SciPy exclusively for the optimization routine: * minimize(method="SLSQP") [0] * A list comprehention which calls ~10-500 pre-fitted PchipInterpolator [1] functions and stores the values as a np.array(). The Pchip functions (and it's first derivatives) are used in the main opt function as well as…

If you find your bottlenecks in SciPy or Numpy, then PyPy will not help. Those are primarily written in C, so the PyPy JIT cannot peer inside and do any magic.

Re: Ask HN: Is anyone using PyPy for real work?

#167

Post by OOPMan »

cool-RR wrote: post #140

Hi Matti. I'm happy to see that you're doing community outreach. I haven't tried PyPy in a while. The general impression I have about PyPy is that as soon as you try to do anything a little bit complicated, things break in unexpected ways and there's little support. Also, I love using Wing IDE for debugging, and if I'm not mistaken it can't debug PyPy code. I'm currently doing multi-agent reinforcement learning resea…

Wow, I didn't know anyone still uses Wing.

The modern debugging tools available in other IDEs work fine with PyPy (and have for years), so I guess that must be a wing issue.

Re: Ask HN: Is anyone using PyPy for real work?

#169

Post by comboy »

mattip wrote: post #119
comboy wrote: post #108

A bit meta. It seems like it would be nice to have no-action tickets for open source projects. Quite often you would want to just thank somebody, or say that you would prefer it that way and don't understand why is it this way or it would be cool to have this or that, but of course opening ticket on github feels like wasting time of the maintainer and especially when you have some feedback like e.g. what would you li…

You can comment on our blog, or open an issue. Frankly, we get so little feedback that dealing with new issues is not a hassle.

I understand, and reaching out like this seems like a great idea and time well spent, I meant it would be nice to have something like this in general, for different. kind of projects.

Re: Ask HN: Is anyone using PyPy for real work?

#170

Post by saltcured »

I used it for real once over a decade ago, when I had to help some researchers who wanted to load an archive of Twitter JSON dumps into an RDBMS. This was basically cleaning/transliterating data fields into CSV that could bulk-import into PostgreSQL. I think we were using Python 2.7 back then.

1. The same naive deserialization and dict processing code ran much faster with PyPy.

2. Conveniently, PyPy also tolerated some broken surrogate pairs in Twitter's UTF8 stream, which threw exceptions when trying to decode the same events with the regular Python interpreter.

I've had some web service code where I wished I could easily swap to PyPy, but these were conservative projects using Apache + mod_wsgi daemons with SE-Linux. If there were a mod_wsgi_pypy that could be a drop-in replacement, I would have advocated for trials/benchmarking with the ops team.

Most other performance-critical work for me has been with combinations of numpy, PyOpenCL, PyOpenGL, and various imaging codecs like `tifffile` or piping numpy arrays in/out of ffmpeg subprocesses.

Post reply on HN