Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Freezing Python’s Dependency Hell

tech.instacart.com

Post reply on HN
Print view View original

41–50 of 152 posts

Re: Freezing Python’s Dependency Hell

#41

Post by jbergknoff »

zedr wrote: post #4

Using a local virtual environment and then building a Docker image removes most of the headaches. I also bundle a Makefile with simple targets. See this as an example: https://github.com/zedr/cffi_test/blob/master/Makefile New projects are created from a template using Cookiecutter. It isn't really so bad in 2018, but I do have a lot of scars from the old days, most of them caused by zc.buildout. The secret is using,…

You can also set a PYTHONUSERBASE environment variable (and `pip install --user`) to scope the installed packages to the project's directory. This is effectively the same as a virtualenv, but doesn't have the requirement on bash or "activation", and it's less magical than virtualenv because these choices are explicit on each command. The tradeoff is that it can be tedious to be explicit, remembering to use `--user` and specify the PYTHONUSERBASE. If you're scripting everything via make, though, then that's not such a burden.

Re: Freezing Python’s Dependency Hell

#42

Post by crdoconnor »

jsmeaton wrote: post #35
crdoconnor wrote: post #28

Earlier quoted context omitted.

A lot of people seem to run into bugs and some hit a brick wall when they report them: https://www.reddit.com/r/Python/comments/8elkqe/pipenv_a_gui... Personally I think poetry doesn't get enough visibility. It's not as hyped as pipenv but it feels a bit nicer: https://poetry.eustace.io/

It really serves a different audience/purpose. Poetry replaces setup.py which is mostly used for building libraries. You still need to create your own virtualenv. Pipenv replaces requirements.txt and handles the virtualenv for you. It can’t be used for packaging libs, but it’s primary purpose is to make developing on apps easier. Lore seems to be much closer to pipenv than poetry.

pyproject.toml and pyproject.lock are analogous to Pipfile and Pipfile.lock

Both attempt to replace requirements.txt, both have dependency resolvers and both are workflow tools.

Re: Freezing Python’s Dependency Hell

#43

Post by zedr »

jbergknoff wrote: post #41
zedr wrote: post #4

Using a local virtual environment and then building a Docker image removes most of the headaches. I also bundle a Makefile with simple targets. See this as an example: https://github.com/zedr/cffi_test/blob/master/Makefile New projects are created from a template using Cookiecutter. It isn't really so bad in 2018, but I do have a lot of scars from the old days, most of them caused by zc.buildout. The secret is using,…

You can also set a PYTHONUSERBASE environment variable (and `pip install --user`) to scope the installed packages to the project's directory. This is effectively the same as a virtualenv, but doesn't have the requirement on bash or "activation", and it's less magical than virtualenv because these choices are explicit on each command. The tradeoff is that it can be tedious to be explicit, remembering to use `--user` a…

Thanks! I didn't know about that. I'll try it out.

Re: Freezing Python’s Dependency Hell

#44

Post by bjpbakker »

I feel that all of these language specific solutions still only solve halve the problem. Your code depends on a lot more than _just_ the python libraries. And often this is exactly what makes projects break on different systems.

Let me make another suggestion: nixpkgs [0] it helps to define exactly that fixed set of dependencies. Not just on published version number, but on the actual source code _and_ all it's dependencies.

[0] - https://nixos.org/nixpkgs/

Re: Freezing Python’s Dependency Hell

#45

Post by spapas82 »

crdoconnor wrote: post #39
spapas82 wrote: post #30

Earlier quoted context omitted.

Is there a reason you don't "activate" your virtualenv? That (with the addition of using mkvirtualenv and friends) is the workflow I use to both dev and prod and am really happy with!

I hate the whole idea of activating virtualenvs. It's a tool that makes it really easy to end up running a command in the wrong environment and see weird behavior instead of a clear error message. I've seen variations on this scenario happen at least 3 times, for instance: 1) Somebody creates script that activates and runs django and commits it. 2) Junior runs script but the virtualenv doesn't get created for some re…

That's 7 characters more you'll need to write all the time! Also you'll need to remember to prepend them to all scrips, ie pip, fab etc. Well that seems to me to be more error prone for juniors than telling them to always use a virtual env (ie have (envnane) in their prompt)!!

Re: Freezing Python’s Dependency Hell

#46

Post by sethgecko »

abhishekjha wrote: post #6

What's wrong with pipenv? I am genuinely curious. On local : mkdir my_project_directory cd my_project_directory export PIPENV_VENV_IN_PROJECT=1 (To make the virtual environment folder determininstic(.venv/) otherwise you will get a hash based directory(my_project_directory-some-hash-value) which might not be suitable for automatic deployments in applications like docker. I don't know why this is not default.) pipenv…

I think you are supposed to use `pipenv sync` on the remote to get the pinned versions from the lock file

Re: Freezing Python’s Dependency Hell

#47

Post by sethgecko »

l0b0 wrote: post #15
abhishekjha wrote: post #6

What's wrong with pipenv? I am genuinely curious. On local : mkdir my_project_directory cd my_project_directory export PIPENV_VENV_IN_PROJECT=1 (To make the virtual environment folder determininstic(.venv/) otherwise you will get a hash based directory(my_project_directory-some-hash-value) which might not be suitable for automatic deployments in applications like docker. I don't know why this is not default.) pipenv…

After having used both, I'm not yet sure which is better of `pipenv` or `pip install --require-hashes` + `python -m venv`. For example, `pipenv sync` doesn't uninstall packages which were previously in the same Pipfile{,.lock}, making the sharing of Pipfile{,.lock} via version control kinda pointless. `PIPENV_VENV_IN_PROJECT` not being the default is also annoying for development.

`pipenv clean` uninstalls all packages not specified in the lock file

Re: Freezing Python’s Dependency Hell

#48

Post by sciurus »

Waterluvian wrote: post #21
tleguijt wrote: post #18

Earlier quoted context omitted.

At most projects we're using pip-tools which generates a fully pinned requirements.txt based on a manually kept (and clean) requirements.in which only contains the specific packages you need without their dependencies

Thanks. I'll investigate this method. It sounds like you hand write dependencies and their versions into the requirements.in file?

The idea is described at https://www.kennethreitz.org/essays/a-better-pip-workflow

Re: Freezing Python’s Dependency Hell

#49

Post by abhishekjha »

yrro wrote: post #11
abhishekjha wrote: post #6

What's wrong with pipenv? I am genuinely curious. On local : mkdir my_project_directory cd my_project_directory export PIPENV_VENV_IN_PROJECT=1 (To make the virtual environment folder determininstic(.venv/) otherwise you will get a hash based directory(my_project_directory-some-hash-value) which might not be suitable for automatic deployments in applications like docker. I don't know why this is not default.) pipenv…

It uses virtualenv, rather than venv. After discovering PYTHONUSERBASE, I no longer need any of the plethora of wrappers around venv/virtualenv.

>After discovering PYTHONUSERBASE, I no longer need any of the plethora of wrappers around venv/virtualenv.

Is there any walkthrough available? Pipenv has one deficiency is that it can be slow at times, particularly when you want to quickly install a dependency and run. Would love to know the alternative.

Re: Freezing Python’s Dependency Hell

#50

Post by ben509 »

sethgecko wrote: post #46
abhishekjha wrote: post #6

What's wrong with pipenv? I am genuinely curious. On local : mkdir my_project_directory cd my_project_directory export PIPENV_VENV_IN_PROJECT=1 (To make the virtual environment folder determininstic(.venv/) otherwise you will get a hash based directory(my_project_directory-some-hash-value) which might not be suitable for automatic deployments in applications like docker. I don't know why this is not default.) pipenv…

I think you are supposed to use `pipenv sync` on the remote to get the pinned versions from the lock file

That's honestly my biggest gripe with pipenv: which command is best for a CI run?

My current magic is:

    pipenv sync $(pipenv --venv > /dev/null || echo '--python 3.6') --dev
The reason is that (magically) adding --python 3.6 will always create a new virtual environment, and I'd rather not do that if the cache is up to date, but running sync by itself won't.

And I think I also want to run `install --deploy`, to check if my Pipfile / lock are in sync or broken.

None of them are huge gripes, more the frustration that it almost works out of the box, but it always seems no one writing these tools ever uses them in prod...

Post reply on HN