Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Freezing Python’s Dependency Hell

tech.instacart.com

Post reply on HN
Print view View original

11–20 of 152 posts

Re: Freezing Python’s Dependency Hell

#11

Post by yrro »

abhishekjha wrote: post #6

What's wrong with pipenv? I am genuinely curious. On local : mkdir my_project_directory cd my_project_directory export PIPENV_VENV_IN_PROJECT=1 (To make the virtual environment folder determininstic(.venv/) otherwise you will get a hash based directory(my_project_directory-some-hash-value) which might not be suitable for automatic deployments in applications like docker. I don't know why this is not default.) pipenv…

It uses virtualenv, rather than venv.

After discovering PYTHONUSERBASE, I no longer need any of the plethora of wrappers around venv/virtualenv.

Re: Freezing Python’s Dependency Hell

#12

Post by ntnn »

Waterluvian wrote: post #9

I ran into a migraine last week: cleaning up requirements.txt How do you determine which requirements are no longer needed when you remove one from your code? In node, your package.json lists only packages YOU installed. So removing them cleans up their dependencies. But in Python, adding one package with pip install might add a dozen entries, none indicating they're dependencies of other packages.

Use requirements.txt volatile.

We use a separate file to list the direct dependencies, 'ddeps.txt' and 'ddeps-dev.txt' for development deps.

Once we update one of these files a clean venv is created, the dependencies installed and the freeze output saved as requirements.txt. Then the dev dependencies are installed and the output of that freeze is saved to requirements-dev.txt.

This preserves the dependencies where we made the conscious choice to require them and also allows us to explicitly vet any new dependencies and versions.

Re: Freezing Python’s Dependency Hell

#13

Post by nickjj »

1. Build Docker image out of requirements.txt

2. Develop application

3. Repeat 1-2 until ready to deploy

4. Run Docker image in production with same dependencies as development

5. ??

6. Profit!

As long as you don't rebuild in between steps 3-4, you'll have the same set of dependencies down to the exact patch level.

Re: Freezing Python’s Dependency Hell

#14

Post by AndrewOMartin »

michaelmcmillan wrote: post #7
Rjevski wrote: post #3

Earlier quoted context omitted.

Pipenv is a good replacement for the above workflow. It manages your dependencies and virtualenvs.

Thanks for the tip, but honestly, I don't need another tool.

I was of the same mentality, until yesterday when I watched this PyCon video, uploaded 13 May 2018.

https://www.youtube.com/watch?v=GBQAKldqgZs

To cut a long story short, if you're happy with virtualenv and pip then that's great, but the idea of pipenv is to replace virtualenv and pip, which means you'll actually have one tool fewer. :)

Re: Freezing Python’s Dependency Hell

#15

Post by l0b0 »

abhishekjha wrote: post #6

What's wrong with pipenv? I am genuinely curious. On local : mkdir my_project_directory cd my_project_directory export PIPENV_VENV_IN_PROJECT=1 (To make the virtual environment folder determininstic(.venv/) otherwise you will get a hash based directory(my_project_directory-some-hash-value) which might not be suitable for automatic deployments in applications like docker. I don't know why this is not default.) pipenv…

After having used both, I'm not yet sure which is better of `pipenv` or `pip install --require-hashes` + `python -m venv`. For example, `pipenv sync` doesn't uninstall packages which were previously in the same Pipfile{,.lock}, making the sharing of Pipfile{,.lock} via version control kinda pointless. `PIPENV_VENV_IN_PROJECT` not being the default is also annoying for development.

Re: Freezing Python’s Dependency Hell

#17

Post by _ikke_ »

yrro wrote: post #11
abhishekjha wrote: post #6

What's wrong with pipenv? I am genuinely curious. On local : mkdir my_project_directory cd my_project_directory export PIPENV_VENV_IN_PROJECT=1 (To make the virtual environment folder determininstic(.venv/) otherwise you will get a hash based directory(my_project_directory-some-hash-value) which might not be suitable for automatic deployments in applications like docker. I don't know why this is not default.) pipenv…

It uses virtualenv, rather than venv. After discovering PYTHONUSERBASE, I no longer need any of the plethora of wrappers around venv/virtualenv.

How do you use PYTHONUSERBASE?

Re: Freezing Python’s Dependency Hell

#18

Post by tleguijt »

Waterluvian wrote: post #9

I ran into a migraine last week: cleaning up requirements.txt How do you determine which requirements are no longer needed when you remove one from your code? In node, your package.json lists only packages YOU installed. So removing them cleans up their dependencies. But in Python, adding one package with pip install might add a dozen entries, none indicating they're dependencies of other packages.

At most projects we're using pip-tools which generates a fully pinned requirements.txt based on a manually kept (and clean) requirements.in which only contains the specific packages you need without their dependencies

Re: Freezing Python’s Dependency Hell

#20

Post by sandGorgon »

genuine question - is nobody using anaconda/conda in production ? I have found the binary install experience in conda far more pleasant than in anything else.

Going forward, the trend is going to be pipenv+manylinux (https://github.com/pypa/manylinux), but conda is super pleasant today

Post reply on HN