Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is anyone using PyPy for real work?

news.ycombinator.com

Post reply on HN
Print view View original

151–160 of 185 posts

Re: Ask HN: Is anyone using PyPy for real work?

#151

Post by martsa1 »

doliveira wrote: post #80
dgroshev wrote: post #78

Earlier quoted context omitted.

At this stage [0], uncompiled native extensions are not yet a bug, but a definite oversight of the maintainer. They should come as precompiled wheels [0]: https://pythonwheels.com

Honestly I don't think I've ever used a precompiled package in Python. Every single C stuff seems to take ages and requires all that fun stuff of installing native system dependencies. Edit: skimming through this page, precompiling seems like an afterthought, and the linked packages don't even seem to mention how to integrate third-party libraries. So I guess I can see why it doesn't deliver on its promises.

Try `--only-binary :all:` to force pip to ignore sdist packages, might help avoid those slow compilations.

Re: Ask HN: Is anyone using PyPy for real work?

#152

Post by tgbugs »

mattip wrote: post #146
tgbugs wrote: post #141

We use pypy3 on musl via gentoo in production to run dataset validation pipelines. The easiest place to see that we use pypy3 is probably [1]. The build process and patches we carry are under [2]. We also use pypy3 to accelerate rdflib parsing and serialization of various RDF formats. See for example [3]. Thanks to you and the whole PyPy team! 1. https://github.com/tgbugs/dockerfiles/blob/6f4ad5d873b7ab267... 2. http…

You're welcome, thanks for sharing. Do you have any numbers about speed vs. an alternative?

I don't have anything rigorous, but I can say that I see the usual ~4x speedup when using rdflib to parse large files so that a 20 minute workload in cpython drops to 4 or 5 minutes when run on pypy3.

I just reran one of my usual benchmarks and I see 2mins for pypy3 (pypy 7.3.12 python 3.10.12) peak memory usage about 8gigs, 4.8mins for python3.11 (3.11.4) peak memory usage about 3.6gigs (2.4x speedup). On another computer running the exact same workload I see 6.3mins and 19mins (3x speedup) with the same peak memory usage.

I don't have any numbers on the dataset pipelines because I never ran them in production on cpython and went straight to pypy3. It is easy for me to switch between the two implementations in this context so I could run a side by side comparison (with the usual caveat that it would be completely non-rigorous).

I also have some internal notes related to a project that I didn't list because it isn't public, isn't in production, and the benchmarks were collected quite a while ago, but I see a 4x increase in throughput when pulling large amounts of data from a postgresql database from 20mbps on cpython 3.6 to 80mbps on pypy3.

Re: Ask HN: Is anyone using PyPy for real work?

#153

Post by CurriedHautious »

cpburns2009 wrote: post #134
CurriedHautious wrote: post #130

What is the compatibility of PyPy with a typical web server deployment? I am currently looking at testing compatibility with Tornado -> SQL Alchemy -> psycopg2. It seems like the C-extensions are a common tripping point. I see the recommendation to use psycopg2cffi, but it seems that package's last release was 2019 :( SQL Alchemy actually points to PyPy in its recommendations of things to try in ORM performance. http…

The only compatibility issue for web development I've run into is database drivers. For PostgreSQL, psycopg2 is not supported. psycopg2cffi is largely unmaintained, and the 2.9.0 version in PyPI lacks some newer features of psycopg2 : the `psycopg2.sql` module and empty result sets raise a RuntimeError in Python 3.7+. The latest commit in on Github does have these changes [1]. Psycopg 3 [2] and pg8000 [3] (as user tl…

Through some back & forth in conda and pip (fighting dependencies), I have been able to get PyPy 3.9 running with my ARM64 Debian system. So far I am seeing a performance decrease up to 2x. I have a series of REST API calls that encapsulate a single DB transaction - a mix of reads and writes. Most of it is leaning on SQL Alchemy, but we have been reaching for psycopg2 for some of the larger insert statements.

I was hoping to see some improvement in ORM performance (SQLAlchemy 1.3) - mainly in the bookkeeping side. Currently the app is about 60% Python app wait time and 40% DB wait time. We have a handful noisy areas which emit a lot of statements (Update 1 row at a time, 10000 times via ORM for example).

I also tried cProfiler to drill down, but as I've seen in Stack Overflow notes that profiler has a larger impact in PyPy over CPython.

Re: Ask HN: Is anyone using PyPy for real work?

#154

Post by CyberDildonics »

ggm wrote: post #12

I'm using pypy to analyse 350m DNS events a day, through python cached dicts to avoid dns lookup stalls. I am getting 95% dict cache hit rate, and use threads with queue locks. Moving to pypy definitely speeded me up a bit. Not as much as I'd hoped, it's probably all about string index into dict and dict management. I may recode into a radix tree. Hard to work out in advance how different it would be: People optimise…

Uplift from normal python was trivial.

By definition if you lift something it is going to go up, but what does this mean?

Re: Ask HN: Is anyone using PyPy for real work?

#155

Post by orf »

doliveira wrote: post #93
toyg wrote: post #83

Earlier quoted context omitted.

Probably a function of the specific set of packages you use, or the pip options you specify. Pretty much all the major C packages come as wheels these days.

They all come as wheels, they just aren't precompiled.

Can you link one that comes as a wheel but is really a source distribution?

Re: Ask HN: Is anyone using PyPy for real work?

#156

Post by probably_a_gpt »

gjvc wrote: post #24
danpalmer wrote: post #18

Earlier quoted context omitted.

Debian is its own worst enemy with things like this. It’s why we eventually moved off it at a previous job, because deploying Python server applications on it was dreadful. I’m sure it’s better if you’re deploying an appliance that you hand off and never touch again, but for evolving modern Python servers it’s not well suited.

Yes 1000x What is it with them which makes them feel entitled to have special "dist-packages" vs "site-packages" as is the default? This drives me nuts, when I have a bunch of native packages I want to bundle in our in-house python deployment. CentOS and Ubuntu are vanilla, and only Debian (mind-boggingly) deviates from the well-trodden path. I still haven't figured out how to beat this dragon. All suggestions welcom…

> I still haven't figured out how to beat this dragon. All suggestions welcome!

Docker

Re: Ask HN: Is anyone using PyPy for real work?

#158

Post by ggm »

syllogism wrote: post #46
ggm wrote: post #12

I'm using pypy to analyse 350m DNS events a day, through python cached dicts to avoid dns lookup stalls. I am getting 95% dict cache hit rate, and use threads with queue locks. Moving to pypy definitely speeded me up a bit. Not as much as I'd hoped, it's probably all about string index into dict and dict management. I may recode into a radix tree. Hard to work out in advance how different it would be: People optimise…

If you have very large dicts, you might find this hash table I wrote for spaCy helpful: https://github.com/explosion/preshed . You need to key the data with 64-bit keys. We use this wrapper around murmurhash for it: https://github.com/explosion/murmurhash There's no docs so obviously this might not be for you. But the software does work, and is efficient. It's been executed many many millions of times now.

I'm in strings, not 64 bit keys. But thanks, nice to share ideas.

Re: Ask HN: Is anyone using PyPy for real work?

#159

Post by ggm »

mattip wrote: post #22
ggm wrote: post #12

I'm using pypy to analyse 350m DNS events a day, through python cached dicts to avoid dns lookup stalls. I am getting 95% dict cache hit rate, and use threads with queue locks. Moving to pypy definitely speeded me up a bit. Not as much as I'd hoped, it's probably all about string index into dict and dict management. I may recode into a radix tree. Hard to work out in advance how different it would be: People optimise…

> it's probably all about string index into dict and dict management Cool. Is the performance here something you would like to pursue? If so could you open an issue [0] with some kind of reproducer? [0] https://foss.heptapod.net/pypy/pypy/-/issues

I'm thinking about how to demonstrate the problem. I have a large pickle but pickle load/dump times across gc.disable()/gc.enable() really doesn't say much.

I need to find out how to instrument the seek/add cost of threads against the shared dict under a lock.

My gut feel is that probably if I inlined things instead of calling out to functions I'd shave a bit more too. So saying "slower than expected" may be unfair because there's limits to how much you can speed this kind of thing up. Thats why I wondered if alternate datastructures were a better fit.

its variable length string indexes into lists/dicts of integer counts. The advantage of a radix trie would be finding the record in semi constant time to the length in bits of the strings, and they do form prefix sets.

Post reply on HN