Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Freezing Python’s Dependency Hell

tech.instacart.com

Post reply on HN
Print view View original

101–110 of 152 posts

Re: Freezing Python’s Dependency Hell

#101

Post by fernandotakai »

RossM wrote: post #91
minitech wrote: post #79

Earlier quoted context omitted.

Doesn't help developers not get different versions of packages. Lockfiles are necessary regardless of Docker.

This is important (though I'm oddly yet to run into this issue with pip; I've only had conflicts with npm and composer before). Freezing dependency sources in Docker images and using (pip install --require-hashes -r requirements.txt) for development seems to cover everything.

yeah, i was going to say the same: i never had that issue in ~7y of python work.

nowadays, requirements + docker solves 99% of everything i do.

maybe it's because i'm not using numpy and the likes?

Re: Freezing Python’s Dependency Hell

#102

Post by coldtea »

throwaway5752 wrote: post #96
coldtea wrote: post #90

Earlier quoted context omitted.

VMs are an easy copout to a problem that shouldn't be a problem in the first place.

That's not true. Python has C libraries, some might need to be built from source, and there's good reason to not allow root access on a lot of systems (and ability to install headers/dev packages, gcc, etc). System package management is hard and coordinating with (ubiqitous, not specific to Python) language package managers magnifies it. Unless you had some other solution in mind that I've missed...

We had the ability to run packages from a custom "root" prefix for ages in UNIX. If only package management tools all worked together, with the same central store, and respecting this...

There's never a need to mess with root access -- not even to install headers/dev packages. Dev tools and compilers can also be made to look to custom locations. I mean, there's never a need aside from self-imposed limitations our OSes and tooling places upon us.

Nothing inherently complex: just tons of accidental complexity.

Those things are only hard because we never did any coordinated effort to fix them.

Re: Freezing Python’s Dependency Hell

#103

Post by ben509 »

luhn wrote: post #93
ben509 wrote: post #50

Earlier quoted context omitted.

That's honestly my biggest gripe with pipenv: which command is best for a CI run? My current magic is: pipenv sync $(pipenv --venv > /dev/null || echo '--python 3.6') --dev The reason is that (magically) adding --python 3.6 will always create a new virtual environment, and I'd rather not do that if the cache is up to date, but running sync by itself won't. And I think I also want to run `install --deploy`, to check i…

I use `pipenv install --deploy --system`. Doesn't create a virtualenv and verifies the lock file. Of course, I definitely understand where you're coming from. It took quite a while for me to figure that out, because --deploy is not well documented.

I'm not sure I want to use --system in CI, just to avoid differences with development machines if possible. That does simplify baking a Docker image, though.

Re: Freezing Python’s Dependency Hell

#104

Post by superbatfish »

virusduck wrote: post #81
sandGorgon wrote: post #20

genuine question - is nobody using anaconda/conda in production ? I have found the binary install experience in conda far more pleasant than in anything else. Going forward, the trend is going to be pipenv+manylinux ( https://github.com/pypa/manylinux ), but conda is super pleasant today

What happens when there isn't a conda recipe for some package or inexplicably some dependency? Do I go back to pip? sudo pip ;) ? Use virtualenv?? Nothing is ever solved.......

Indeed, using pip from within a conda environment is trivial.

But the preferred solution is to make a conda package for yourself, and it's really quite simple. You can host it from anaconda.org, or from a local directory for crying out loud.

Re: Freezing Python’s Dependency Hell

#105

Post by hmaarrfk »

peatmoss wrote: post #70
pytyper2 wrote: post #53

I'm not sure why the scientists don't use VMs and simply save the virtual disk files? That would at the very least allow them to verify the settings at a later date. Fresh install reproducibility doesn't seem necessary to verify experimental findings as long as the original vm is available to boot up.

My guesses are that: 1. Integrating the development environment on their host PC (for example connecting RStudio in R's case, or connecting their web browser back to a server running in the VM in the case of Jupyter) is another set of skills to master. 2. Many data analyses are memory hungry unless you want to resort to coding practices that optimize for memory consumption. The overhead of running a VM is a bummer fo…

Often scientists are using hardware to acquire new data. The acquisition hardware might be on a PC that came installed from the manufacturer where you are told not to change anything.

Touching that PC, in anyway would be considered harmful to everybody using that specific piece of equipment.

Therefore, from the beginning of your acquisition, you are basically using a machine you don't control.

Re: Freezing Python’s Dependency Hell

#106

Post by hmaarrfk »

throwaway5752 wrote: post #96
coldtea wrote: post #90

Earlier quoted context omitted.

VMs are an easy copout to a problem that shouldn't be a problem in the first place.

That's not true. Python has C libraries, some might need to be built from source, and there's good reason to not allow root access on a lot of systems (and ability to install headers/dev packages, gcc, etc). System package management is hard and coordinating with (ubiqitous, not specific to Python) language package managers magnifies it. Unless you had some other solution in mind that I've missed...

conda basically proves that you can install almost everything one needs in the user's home directory. they have been working more and more on being completely independent from things like system compilers as well.

Re: Freezing Python’s Dependency Hell

#108

Post by akarambir »

abhishekjha wrote: post #6

What's wrong with pipenv? I am genuinely curious. On local : mkdir my_project_directory cd my_project_directory export PIPENV_VENV_IN_PROJECT=1 (To make the virtual environment folder determininstic(.venv/) otherwise you will get a hash based directory(my_project_directory-some-hash-value) which might not be suitable for automatic deployments in applications like docker. I don't know why this is not default.) pipenv…

Pycharm just released 2018.2 with support for pipenv :) https://www.jetbrains.com/pycharm/whatsnew/#v2018-2-python

Re: Freezing Python’s Dependency Hell

#110

Post by gvalkov »

jbergknoff wrote: post #41
zedr wrote: post #4

Using a local virtual environment and then building a Docker image removes most of the headaches. I also bundle a Makefile with simple targets. See this as an example: https://github.com/zedr/cffi_test/blob/master/Makefile New projects are created from a template using Cookiecutter. It isn't really so bad in 2018, but I do have a lot of scars from the old days, most of them caused by zc.buildout. The secret is using,…

You can also set a PYTHONUSERBASE environment variable (and `pip install --user`) to scope the installed packages to the project's directory. This is effectively the same as a virtualenv, but doesn't have the requirement on bash or "activation", and it's less magical than virtualenv because these choices are explicit on each command. The tradeoff is that it can be tedious to be explicit, remembering to use `--user` a…

There is no need to activate a virtualenv to use it. Just call $VIRTUALENV/bin/python directly. Activating is just a convenience for doing interactive work.
Post reply on HN