Ask HN: Is anyone using PyPy for real work? • Hacker News Board

Re: Ask HN: Is anyone using PyPy for real work?

#81

Post by PaulHoule » Mon, Jul 31, 2023, 2:38 PM UTC

mattip wrote: post #10

PaulHoule wrote: post #6

I use CPython most of the time but PyPy was a real lifesaver when I was doing a project that bridged EMOF and RDF, particularly I was working with moderately sized RDF models (say 10 million triples) with rdflib. With CPython, I was frustrated with how slow it was, and complained about it to the people I was working with, PyPy was a simple upgrade that sped up my code to the point where it was comfortable to work wit…

Is your group still using it?

That particular code has been retired because after a quite a bit of trying things that weren’t quite right we understood the problem and found a better way to do it. I’m doing the next round of related work (logically modeling XSLT schemas and associated messages in OWL) in Java because there is already a library that almost does was I want.

I am still using this library that I wrote

https://paulhoule.github.io/gastrodon/

to visualize RDF data so even if I make my RDF model in Java I am likely to load it up in Python to explore it. I don’t know if they are using PyPy but there is at least one big bank that has people using Gastrodon for the same purpose.

Re: Ask HN: Is anyone using PyPy for real work?

#82

Post by m_antis89 » Mon, Jul 31, 2023, 2:44 PM UTC

> Is anyone using PyPy for real work? Yes.

Re: Ask HN: Is anyone using PyPy for real work?

#83

Post by toyg » Mon, Jul 31, 2023, 2:46 PM UTC

doliveira wrote: post #80

dgroshev wrote: post #78

Earlier quoted context omitted.

At this stage [0], uncompiled native extensions are not yet a bug, but a definite oversight of the maintainer. They should come as precompiled wheels [0]: https://pythonwheels.com

Honestly I don't think I've ever used a precompiled package in Python. Every single C stuff seems to take ages and requires all that fun stuff of installing native system dependencies. Edit: skimming through this page, precompiling seems like an afterthought, and the linked packages don't even seem to mention how to integrate third-party libraries. So I guess I can see why it doesn't deliver on its promises.

Probably a function of the specific set of packages you use, or the pip options you specify. Pretty much all the major C packages come as wheels these days.

Re: Ask HN: Is anyone using PyPy for real work?

#84

Post by radus » Mon, Jul 31, 2023, 2:47 PM UTC

I don’t use it because I make frequent use of scientific libraries. If it were possible to use on a function by function basis, with a decorator like numba, I would definitely give it a go.

Re: Ask HN: Is anyone using PyPy for real work?

#85

Post by dgroshev » Mon, Jul 31, 2023, 2:50 PM UTC

doliveira wrote: post #80

dgroshev wrote: post #78

Earlier quoted context omitted.

At this stage [0], uncompiled native extensions are not yet a bug, but a definite oversight of the maintainer. They should come as precompiled wheels [0]: https://pythonwheels.com

Honestly I don't think I've ever used a precompiled package in Python. Every single C stuff seems to take ages and requires all that fun stuff of installing native system dependencies. Edit: skimming through this page, precompiling seems like an afterthought, and the linked packages don't even seem to mention how to integrate third-party libraries. So I guess I can see why it doesn't deliver on its promises.

You can try pip install pillow for a good example of how it works. I suspect there's a strong survivorship bias here, as you'd only notice the packages that don't ship with wheels.

Re: Ask HN: Is anyone using PyPy for real work?

#86

Post by Twirrim » Mon, Jul 31, 2023, 2:55 PM UTC

andrewstuart wrote: post #8

I’ve been aware of it for a long time. I don’t use it. Why would I use it, what’s the compelling benefit?

It's a python runtime that contains a JIT, as a result it can be phenomenally faster. Like with any JITted runtime, it depends a bit on what your code is doing, and how long you're running it for as there is a little (but honestly very little) bit of up front overhead.

Re: Ask HN: Is anyone using PyPy for real work?

#87

Post by sitkack » Mon, Jul 31, 2023, 2:58 PM UTC

ggm wrote: post #12

I'm using pypy to analyse 350m DNS events a day, through python cached dicts to avoid dns lookup stalls. I am getting 95% dict cache hit rate, and use threads with queue locks. Moving to pypy definitely speeded me up a bit. Not as much as I'd hoped, it's probably all about string index into dict and dict management. I may recode into a radix tree. Hard to work out in advance how different it would be: People optimise…

One should really consider using containers in this situation.

Re: Ask HN: Is anyone using PyPy for real work?

#88

Post by ofek » Mon, Jul 31, 2023, 2:58 PM UTC

bofaGuy wrote: post #52

My biggest issue is that DataDog doesn’t support PyPy. Out of curiosity, I made a new branch of our app and took out DataDog and observed a significant improvement in performance when using PyPy vs CPython on the same branch (but can’t remember how much).

Do you mean the Python tracing library does not work out-of-the-box?

disclaimer: I work there but not on the APM team

Re: Ask HN: Is anyone using PyPy for real work?

#89

Post by kevin_thibedeau » Mon, Jul 31, 2023, 2:58 PM UTC

gjvc wrote: post #24

danpalmer wrote: post #18

Earlier quoted context omitted.

Debian is its own worst enemy with things like this. It’s why we eventually moved off it at a previous job, because deploying Python server applications on it was dreadful. I’m sure it’s better if you’re deploying an appliance that you hand off and never touch again, but for evolving modern Python servers it’s not well suited.

Yes 1000x What is it with them which makes them feel entitled to have special "dist-packages" vs "site-packages" as is the default? This drives me nuts, when I have a bunch of native packages I want to bundle in our in-house python deployment. CentOS and Ubuntu are vanilla, and only Debian (mind-boggingly) deviates from the well-trodden path. I still haven't figured out how to beat this dragon. All suggestions welcom…

Dist packages is the right way to handle Python libs. You'd prefer to have the distro package manager clashing with Pip? Never knowing who installed what. Breaking things when updates are made.

Re: Ask HN: Is anyone using PyPy for real work?

#90

Post by Twirrim » Mon, Jul 31, 2023, 3:00 PM UTC

I was experimenting with some dynamic programming 0/1 knapsack code last week. PyPy available through the distro (7.3.9) was making a reasonable speed up, but not phenomenally. Out of curiousity I grabbed the latest version through pyenv (7.3.12) and it looks like some changes between them suddenly had the code sit in a sweet spot with it, I saw a couple of orders of magnitude better performance out of it. Good work.

I'm rarely using python in places at work where it would suit it (lots of python usage, but they're more on the order of short run tools), but I'm always looking for chances and always using it for random little personal things.