Live data from Hacker News

Ask HN: Is anyone using PyPy for real work?

news.ycombinator.com

91–100 of 185 posts

Re: Ask HN: Is anyone using PyPy for real work?

#91
Thanks for reminding me to look at PyPy again. I usually start all my new Python projects with this block of commands that I keep handy:

Create venv and activate it and install packages:

  python3 -m venv venv
  source venv/bin/activate
  python3 -m pip install --upgrade pip
  python3 -m pip install wheel
  pip install -r requirements.txt

I wanted a similar one-liner that I could use on a fresh Ubuntu machine so I can try out PyPy easily in the same way. After a bit of fiddling, I came up with this monstrosity which should work with both bash and zsh (though I only tested it on zsh):

Create venv and activate it and install packages using pyenv/pypy/pip:

  if [ -d "$HOME/.pyenv" ]; then rm -Rf $HOME/.pyenv; fi && \
  curl https://pyenv.run | bash && \
  DEFAULT_SHELL=$(basename "$SHELL") && \
  if [ "$DEFAULT_SHELL" = "zsh" ]; then RC_FILE=~/.zshrc; else RC_FILE=~/.bashrc; fi && \
  if ! grep -q 'export PATH="$HOME/.pyenv/bin:$PATH"' $RC_FILE; then echo -e '\nexport PATH="$HOME/.pyenv/bin:$PATH"' >> $RC_FILE; fi && \
  if ! grep -q 'eval "$(pyenv init -)"' $RC_FILE; then echo 'eval "$(pyenv init -)"' >> $RC_FILE; fi && \
  if ! grep -q 'eval "$(pyenv virtualenv-init -)"' $RC_FILE; then echo 'eval "$(pyenv virtualenv-init -)"' >> $RC_FILE; fi && \
  source $RC_FILE && \
  LATEST_PYPY=$(pyenv install --list | grep -P '^  pypy[0-9\.]*-\d+\.\d+' | grep -v -- '-src' | tail -1) && \
  LATEST_PYPY=$(echo $LATEST_PYPY | tr -d '[:space:]') && \
  echo "Installing PyPy version: $LATEST_PYPY" && \
  pyenv install $LATEST_PYPY && \
  pyenv local $LATEST_PYPY && \
  pypy -m venv venv && \
  source venv/bin/activate && \
  pip install --upgrade pip && \
  pip install wheel && \
  pip install -r requirements.txt
Maybe others will find it useful.

Re: Ask HN: Is anyone using PyPy for real work?

#92
post #29

I use it for data transformation, cleanup and enrichment. (TXT, CSV, Json, XML, database) to (TXT, CSV, JSON, XML, database). Speed up of 30x - 40x. The highest speedup on those that require logic in the transformation. (lot of function calls, numerical operations and dictionary lookups).

Similar. I was working on some ETL work with SQLite, and now PyPy is my regular tool for getting better performance at similar jobs.

Same. I have used it for many ETL jobs, usually with about a 10x speed up. It also pulled in the latency on some Flask rest apis.

Re: Ask HN: Is anyone using PyPy for real work?

#93
post #83

Earlier quoted context omitted.

Honestly I don't think I've ever used a precompiled package in Python. Every single C stuff seems to take ages and requires all that fun stuff of installing native system dependencies. Edit: skimming through this page, precompiling seems like an afterthought, and the linked packages don't even seem to mention how to integrate third-party libraries. So I guess I can see why it doesn't deliver on its promises.

Probably a function of the specific set of packages you use, or the pip options you specify. Pretty much all the major C packages come as wheels these days.

They all come as wheels, they just aren't precompiled.

Re: Ask HN: Is anyone using PyPy for real work?

#94
post #83

Earlier quoted context omitted.

Probably a function of the specific set of packages you use, or the pip options you specify. Pretty much all the major C packages come as wheels these days.

They all come as wheels, they just aren't precompiled.

I honestly can't remember the last time I had to compile anything, and I am on Windows.

Re: Ask HN: Is anyone using PyPy for real work?

#95

Earlier quoted context omitted.

Honestly I don't think I've ever used a precompiled package in Python. Every single C stuff seems to take ages and requires all that fun stuff of installing native system dependencies. Edit: skimming through this page, precompiling seems like an afterthought, and the linked packages don't even seem to mention how to integrate third-party libraries. So I guess I can see why it doesn't deliver on its promises.

You can try pip install pillow for a good example of how it works. I suspect there's a strong survivorship bias here, as you'd only notice the packages that don't ship with wheels.

Yeah, perhaps. One I remember from last year is the cryptography and numpy package, for instance. Now they do seem to ship with binary wheels, at least for my current Python and Linux version.

Kerberos and Hadoop stuff obviously still doesn't, though. I guess the joke's on me for being stuck in this stack...

Re: Ask HN: Is anyone using PyPy for real work?

#96
I do think it would be very useful to have an online tool that lets you paste in your requirements.txt and then tells you which of the libraries have been recently verified to work properly with PyPy without a lot of additional fuss.

Also, you might want to flag the libraries that technically "work" but still require an extremely long and involved build process. For example, I recently started the process of installing Pandas with pip in a PyPy venv and it was stuck on `Getting requirements to build wheel ...` for a very long time, like 20+ minutes.

Re: Ask HN: Is anyone using PyPy for real work?

#97
I actually donated to the Pypy project in the past but I don’t use it.

Two reasons for my hesitation:

1) Cpython is fast enough for most things I need to do. The speed improvement from Pypy is either not enough or not necessary.

2) Lingering doubts about subtle incompatibility (in terms of library support) that I might have to spend hours getting to the bottom of.

I already work long hours and don’t have bandwidth to tinker. With Cpython, although slow, I can be assured is the standard surface that everyone targets, and I can google solutions for.

It’s the subtle things that i waste a lot of time on. It’s analogous to an Ubuntu user trying to use Red Hat. They’re both Linuxes but the way things are done are different enough that they trip you up.

The only way to get out of this quandary is for Pypy to be a first class citizen. Guido will never endorse this so this means a bunch of us will always have hesitation putting it into production systems.

Re: Ask HN: Is anyone using PyPy for real work?

#98
post #72
post #2

You should probably put "Ask HN:" in your title. Personally I don't use PyPy for anything, though I have followed it with interest. Most of the things I need to go faster are numerical, so Numba and Cython seem more appropriate.

Cut him some slack, he's only been registered for 10 years

I don’t think it’s about being strict or condescending. In some HN readers the post will show up in a different catalogue and generally be easier for people to find, thus giving the post more visibility :)

Edit; typo

Re: Ask HN: Is anyone using PyPy for real work?

#99
post #87
post #12

I'm using pypy to analyse 350m DNS events a day, through python cached dicts to avoid dns lookup stalls. I am getting 95% dict cache hit rate, and use threads with queue locks. Moving to pypy definitely speeded me up a bit. Not as much as I'd hoped, it's probably all about string index into dict and dict management. I may recode into a radix tree. Hard to work out in advance how different it would be: People optimise…

One should really consider using containers in this situation.

Can you describe what in this situation warrants it?

I'm very curious about where the line is/should be.

Re: Ask HN: Is anyone using PyPy for real work?

#100
post #12

I'm using pypy to analyse 350m DNS events a day, through python cached dicts to avoid dns lookup stalls. I am getting 95% dict cache hit rate, and use threads with queue locks. Moving to pypy definitely speeded me up a bit. Not as much as I'd hoped, it's probably all about string index into dict and dict management. I may recode into a radix tree. Hard to work out in advance how different it would be: People optimise…

Debian is its own worst enemy with things like this. It’s why we eventually moved off it at a previous job, because deploying Python server applications on it was dreadful. I’m sure it’s better if you’re deploying an appliance that you hand off and never touch again, but for evolving modern Python servers it’s not well suited.

What distro did you move to? IME debian as a base image for python app containers is also kind of a pain.
Post reply on HN