Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Freezing Python’s Dependency Hell

tech.instacart.com

Post reply on HN
Print view View original

21–30 of 152 posts

Re: Freezing Python’s Dependency Hell

#21

Post by Waterluvian »

tleguijt wrote: post #18
Waterluvian wrote: post #9

I ran into a migraine last week: cleaning up requirements.txt How do you determine which requirements are no longer needed when you remove one from your code? In node, your package.json lists only packages YOU installed. So removing them cleans up their dependencies. But in Python, adding one package with pip install might add a dozen entries, none indicating they're dependencies of other packages.

At most projects we're using pip-tools which generates a fully pinned requirements.txt based on a manually kept (and clean) requirements.in which only contains the specific packages you need without their dependencies

Thanks. I'll investigate this method. It sounds like you hand write dependencies and their versions into the requirements.in file?

Re: Freezing Python’s Dependency Hell

#22

Post by koolba »

nickjj wrote: post #13

1. Build Docker image out of requirements.txt 2. Develop application 3. Repeat 1-2 until ready to deploy 4. Run Docker image in production with same dependencies as development 5. ?? 6. Profit! As long as you don't rebuild in between steps 3-4, you'll have the same set of dependencies down to the exact patch level.

Bingo. If you’re not vendoring the binaries of your dependencies as part of a release then you’re doing it wrong.

It doesn’t have to be docker, containers just makes it easy to have immutable snapshots. Anything that packages it all up (including a simple tarball) is enough.

Re: Freezing Python’s Dependency Hell

#23

Post by NoGravitas »

abhishekjha wrote: post #6

What's wrong with pipenv? I am genuinely curious. On local : mkdir my_project_directory cd my_project_directory export PIPENV_VENV_IN_PROJECT=1 (To make the virtual environment folder determininstic(.venv/) otherwise you will get a hash based directory(my_project_directory-some-hash-value) which might not be suitable for automatic deployments in applications like docker. I don't know why this is not default.) pipenv…

As far as I can tell, the main difference is that this also uses pyenv to manage python versions separate from system python packages. There was an article a couple of weeks ago about combining pyenv + pipenv, and this doesn't really seem to add anything over that combination except an opinionated wrapper script.

Re: Freezing Python’s Dependency Hell

#24

Post by chocks »

We’ve recently went through this process at our company & chose to use pipenv as the dependency management tool. As mentioned in the article, pipenv is under active development but takes care of many things that we had custom scripts before such as requirements hashs, in-built graph of dependencies, automatic retries of failed dependencies, automatic re-ordering of dependency installations etc. it also has a few quirks - we had to pick a version that had most commands working & also pipenv install is painfully slow & didn’t seem to have a caching strategy for already built virtualenvs.

Re: Freezing Python’s Dependency Hell

#25

Post by striking »

neuland wrote: post #10
Waterluvian wrote: post #9

I ran into a migraine last week: cleaning up requirements.txt How do you determine which requirements are no longer needed when you remove one from your code? In node, your package.json lists only packages YOU installed. So removing them cleans up their dependencies. But in Python, adding one package with pip install might add a dozen entries, none indicating they're dependencies of other packages.

I’m not sure about other people, but that is how I use requirements.txt. You don’t have to dump the entire output of pip freeze in there. You can just list the dependencies you want.

Or you can list direct dependencies in another file and regenerate requirements.txt with `pip freeze` whenever you change the other file. Especially easy with Make.

Re: Freezing Python’s Dependency Hell

#26

Post by BerislavLopac »

Since we're sharing XKCD cartoons, here's one that comes to mind: https://xkcd.com/927/

So not to disappoint, here's another contestant: Poetry [0]

That said, in my experience it works best if don't force any particular workflow on your developers, but maintain a solid and repeatable process for testing and deployment. People have different mental models of their development environments -- I personally use virtualfish (or virtualenvwrapper if I'm on Bash), while a colleague works with `python -m venv`; and we have played with pipenv, pyenv, anaconda and poetry in various cases.

As long as your requirements are clearly defined -- requirements.txt works perfectly well for applications, and setup.py for libraries [1] -- any method should be good enough to build a development environment. On the other hand, your integration, testing and deployment process should be universal, and fully automated if possible, and of course independent of any developer's environment.

[0] https://github.com/sdispater/poetry

[1] https://caremad.io/posts/2013/07/setup-vs-requirement/

Re: Freezing Python’s Dependency Hell

#27

Post by jessaustin »

I bitch a lot about npm, but then I remember that time when python's package distribution drove me to learn a new language. I can't help but notice that TFA and all the comments here are only talking about one end of this: managing your dev environment. Is there a similar work explaining how to distribute python packages in a straightforward manner? Is that article compatible with this one?

Re: Freezing Python’s Dependency Hell

#28

Post by crdoconnor »

abhishekjha wrote: post #6

What's wrong with pipenv? I am genuinely curious. On local : mkdir my_project_directory cd my_project_directory export PIPENV_VENV_IN_PROJECT=1 (To make the virtual environment folder determininstic(.venv/) otherwise you will get a hash based directory(my_project_directory-some-hash-value) which might not be suitable for automatic deployments in applications like docker. I don't know why this is not default.) pipenv…

A lot of people seem to run into bugs and some hit a brick wall when they report them:

https://www.reddit.com/r/Python/comments/8elkqe/pipenv_a_gui...

Personally I think poetry doesn't get enough visibility. It's not as hyped as pipenv but it feels a bit nicer:

https://poetry.eustace.io/

Re: Freezing Python’s Dependency Hell

#30

Post by spapas82 »

michaelmcmillan wrote: post #2

Never had a problem with dependencies in Python. Just keep it simple. When starting a new project: virtualenv venv -p *path-to-python-version-you-want* ./venv/bin/pip install *name-of-package* When running that project: ./venv/bin/python *name-of-python-file* Many people don't realize that the venv/bin/ contains all the relevant binaries with the right library path's out of the box.

Is there a reason you don't "activate" your virtualenv?

That (with the addition of using mkvirtualenv and friends) is the workflow I use to both dev and prod and am really happy with!

Post reply on HN