Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is anyone using PyPy for real work?

news.ycombinator.com

Post reply on HN
Print view View original

141–150 of 185 posts

Re: Ask HN: Is anyone using PyPy for real work?

#141

Post by tgbugs »

We use pypy3 on musl via gentoo in production to run dataset validation pipelines. The easiest place to see that we use pypy3 is probably [1]. The build process and patches we carry are under [2].

We also use pypy3 to accelerate rdflib parsing and serialization of various RDF formats. See for example [3].

Thanks to you and the whole PyPy team!

1. https://github.com/tgbugs/dockerfiles/blob/6f4ad5d873b7ab267...

2. https://github.com/tgbugs/dockerfiles/blob/6f4ad5d873b7ab267...

3. https://github.com/SciCrunch/sparc-curation/blob/0fdf393e26f...

Re: Ask HN: Is anyone using PyPy for real work?

#142

Post by stefanor »

gjvc wrote: post #24
danpalmer wrote: post #18

Earlier quoted context omitted.

Debian is its own worst enemy with things like this. It’s why we eventually moved off it at a previous job, because deploying Python server applications on it was dreadful. I’m sure it’s better if you’re deploying an appliance that you hand off and never touch again, but for evolving modern Python servers it’s not well suited.

Yes 1000x What is it with them which makes them feel entitled to have special "dist-packages" vs "site-packages" as is the default? This drives me nuts, when I have a bunch of native packages I want to bundle in our in-house python deployment. CentOS and Ubuntu are vanilla, and only Debian (mind-boggingly) deviates from the well-trodden path. I still haven't figured out how to beat this dragon. All suggestions welcom…

> What is it with them which makes them feel entitled to have special "dist-packages" vs "site-packages" as is the default? This drives me nuts, when I have a bunch of native packages I want to bundle in our in-house python deployment. CentOS and Ubuntu are vanilla, and only Debian (mind-boggingly) deviates from the well-trodden path.

Hi, I'm one of the people that look after this bit of Debian (and it's exactly the same in Ubuntu, FWIW).

It's like that to solve a problem (of course, everything has a reason). The idea is that Debian provides a Python that's deeply integrated into Debian packages. But if you want to build your own Python from source, you can. What you build will use site-packages, so it won't have any overlap with Debian's Python.

Unfortunately, while this approach was designed to be something all package-managed distributions could do, nobody else has adopted it, and consequently the code to make it work has never been pushed upstream. So, it's left as a Debian/Ubuntu oddity that confuses people. Sorry about that.

My recommendations are: 1. If you want more control over your Python than you get from Debian's package-managed python, build your own from source (or use a docker image that does that). 2. Deploy your apps with virtualenvs or system-level containers per app.

Re: Ask HN: Is anyone using PyPy for real work?

#143

Post by stefanor »

eigenvalue wrote: post #91

Thanks for reminding me to look at PyPy again. I usually start all my new Python projects with this block of commands that I keep handy: Create venv and activate it and install packages: python3 -m venv venv source venv/bin/activate python3 -m pip install --upgrade pip python3 -m pip install wheel pip install -r requirements.txt I wanted a similar one-liner that I could use on a fresh Ubuntu machine so I can try out…

If you have a system-level installed pypy, the pypy equivalent is:

  python3 -m venv -p pypy3 venv
  source venv/bin/activate
  python3 -m pip install --upgrade pip
  python3 -m pip install wheel
  pip install -r requirements.txt
Not very different...

Re: Ask HN: Is anyone using PyPy for real work?

#144

Post by KingMachiavelli »

gjvc wrote: post #24
danpalmer wrote: post #18

Earlier quoted context omitted.

Debian is its own worst enemy with things like this. It’s why we eventually moved off it at a previous job, because deploying Python server applications on it was dreadful. I’m sure it’s better if you’re deploying an appliance that you hand off and never touch again, but for evolving modern Python servers it’s not well suited.

Yes 1000x What is it with them which makes them feel entitled to have special "dist-packages" vs "site-packages" as is the default? This drives me nuts, when I have a bunch of native packages I want to bundle in our in-house python deployment. CentOS and Ubuntu are vanilla, and only Debian (mind-boggingly) deviates from the well-trodden path. I still haven't figured out how to beat this dragon. All suggestions welcom…

IMO bespoke containers using whatever python package manager makes sense for each project. Or make the leap to Nix(OS) and then still have to force every python project into compliance which can be very easy if the PyPy packages you need are already in the main Nix repo (nixpkgs) or very difficult if depends on a lot of uncommon packages, uses poetry, etc.

Since PEP 665 was rejected the Python ecosystem continues to lack a reasonable package manager and the lack of hashed based lock files prevents building on top of the current python project/package managers.

Re: Ask HN: Is anyone using PyPy for real work?

#145

Post by deizel »

eigenvalue wrote: post #91

Thanks for reminding me to look at PyPy again. I usually start all my new Python projects with this block of commands that I keep handy: Create venv and activate it and install packages: python3 -m venv venv source venv/bin/activate python3 -m pip install --upgrade pip python3 -m pip install wheel pip install -r requirements.txt I wanted a similar one-liner that I could use on a fresh Ubuntu machine so I can try out…

Given you'll want to activate a virtual environment for most Python projects, and projects live in directories.. I find myself constantly reaching for direnv. https://github.com/direnv/direnv/wiki/Python

    echo "layout python\npip install --upgrade pip pip-tools setuptools wheel\npip-sync" > .envrc
When you CD into a given project, it'll activate the venv, upgrade to non-ancient versions of Pip/etc with support for latest PEPs (ie. `pyproject.toml` support on new Python 3.9 env), verify the latest pinned packages are present.. it's just too useful not to have.

    direnv stdlib
This command (or this link https://direnv.net/man/direnv-stdlib.1.html) will print many useful functions that can be used in the `.envrc` shell script that is loaded when entering directories, ranging from many languages, to `dotenv` support, to `on_git_branch` for e.g. syncing deps when switching feature branches.

Check it out if you haven't.. I've been using it for more years than I can count and being able to CD from a PHP project to a Ruby project to a Python project with ease really helps with context switching.

Re: Ask HN: Is anyone using PyPy for real work?

#146

Post by mattip »

tgbugs wrote: post #141

We use pypy3 on musl via gentoo in production to run dataset validation pipelines. The easiest place to see that we use pypy3 is probably [1]. The build process and patches we carry are under [2]. We also use pypy3 to accelerate rdflib parsing and serialization of various RDF formats. See for example [3]. Thanks to you and the whole PyPy team! 1. https://github.com/tgbugs/dockerfiles/blob/6f4ad5d873b7ab267... 2. http…

You're welcome, thanks for sharing. Do you have any numbers about speed vs. an alternative?

Re: Ask HN: Is anyone using PyPy for real work?

#148

Post by gjvc »

rlpb wrote: post #64
smashed wrote: post #49

Earlier quoted context omitted.

You're not really answering why they are important? Is it because .deb packages will install inside dist-packages and when you run pip install as root without a virtual env, it installs inside site-packages? I don't really see how this helps though? Sure you won't get paths to clash between the two but you still have duplicate packages which is probably not what you want..

Debian ships packages with a coherent dependency structure that crosses language boundaries. You don't need to care what language something is written in to be able to "apt install" it. The expectation is that if it "apt installed" then it should Just Work because all the required dependencies were also pulled in from Debian at the same time. Debian also tries to ship just one version of everything in a single distri…

OK, but... I get the same problem when I compile python from source. I'm not talking about the distribution's base files. In fact, I am in the business of creating an /opt/gjvc-corp-tools/ prefix with all my packages under there. When I compile python from source on Debian, the resulting installation (from make install) does not have a site-packages directory in-place already. That is what is mind-boggling.

Re: Ask HN: Is anyone using PyPy for real work?

#149

Post by lozenge »

ADcorpo wrote: post #9

This post is a funny coincidence as I tried today to speed-up a CI pipeline running ~10k tests with pytest by switching to pypy. I am still working on it but the main issue is psycopg support for now, as I had to install psycopg2cffi in my test environment, but it will probably prevent me from using pypy for running our test suite, because psycopg2cffi does not have the same features and versions as psycopg2. This me…

One compromise could be to run pypy on draft PRs and CPython on approved PRs and master?

Re: Ask HN: Is anyone using PyPy for real work?

#150

Post by pmontra »

rlpb wrote: post #64
smashed wrote: post #49

Earlier quoted context omitted.

You're not really answering why they are important? Is it because .deb packages will install inside dist-packages and when you run pip install as root without a virtual env, it installs inside site-packages? I don't really see how this helps though? Sure you won't get paths to clash between the two but you still have duplicate packages which is probably not what you want..

Debian ships packages with a coherent dependency structure that crosses language boundaries. You don't need to care what language something is written in to be able to "apt install" it. The expectation is that if it "apt installed" then it should Just Work because all the required dependencies were also pulled in from Debian at the same time. Debian also tries to ship just one version of everything in a single distri…

> This is fundamentally at odds with pip

It's at odds with everything. I leave the system versions of any language alone and use language manager tools or docker to be able to run the exact version that any project of my customers require. Asdf is my favorite because it handles nearly everything, even PostgreSQL.

Post reply on HN