Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Freezing Python’s Dependency Hell

tech.instacart.com

Post reply on HN
Print view View original

91–100 of 152 posts

Re: Freezing Python’s Dependency Hell

#91

Post by RossM »

minitech wrote: post #79
nickjj wrote: post #13

1. Build Docker image out of requirements.txt 2. Develop application 3. Repeat 1-2 until ready to deploy 4. Run Docker image in production with same dependencies as development 5. ?? 6. Profit! As long as you don't rebuild in between steps 3-4, you'll have the same set of dependencies down to the exact patch level.

Doesn't help developers not get different versions of packages. Lockfiles are necessary regardless of Docker.

This is important (though I'm oddly yet to run into this issue with pip; I've only had conflicts with npm and composer before). Freezing dependency sources in Docker images and using (pip install --require-hashes -r requirements.txt) for development seems to cover everything.

Re: Freezing Python’s Dependency Hell

#93

Post by luhn »

ben509 wrote: post #50
sethgecko wrote: post #46

Earlier quoted context omitted.

I think you are supposed to use `pipenv sync` on the remote to get the pinned versions from the lock file

That's honestly my biggest gripe with pipenv: which command is best for a CI run? My current magic is: pipenv sync $(pipenv --venv > /dev/null || echo '--python 3.6') --dev The reason is that (magically) adding --python 3.6 will always create a new virtual environment, and I'd rather not do that if the cache is up to date, but running sync by itself won't. And I think I also want to run `install --deploy`, to check i…

I use `pipenv install --deploy --system`. Doesn't create a virtualenv and verifies the lock file.

Of course, I definitely understand where you're coming from. It took quite a while for me to figure that out, because --deploy is not well documented.

Re: Freezing Python’s Dependency Hell

#94

Post by Animats »

Use a fresh virtualenv for each project

As a form of version pinning, this locks in old versions and creates technical debt. A few years downstream, you're locked into library modules no longer supported and years behind in bug fixes.

Re: Freezing Python’s Dependency Hell

#95

Post by breatheoften »

michaelmcmillan wrote: post #33
spapas82 wrote: post #30

Earlier quoted context omitted.

Is there a reason you don't "activate" your virtualenv? That (with the addition of using mkvirtualenv and friends) is the workflow I use to both dev and prod and am really happy with!

I don't like "magic". I don't need anything to hijack PS1 and muck around with my shell.

I heartily agree with this. I really dislike the tools that provide their functionality by mucking around with the shell environment in (essentially random after accounting for platform variations) ways ...

Tools like nvm, rvm, ros ... If I can use a solution for managing a development context that doesn't involve mucking around with the shell environment I much prefer it. Configuration via the sourcing of shell scripts is a very fragile interface, doesn't work when with good (ie non-bash) shell, and almost always eventually leads to bugs when some workflow triggers processes in a manner that fails to inherit the shell environment...

Re: Freezing Python’s Dependency Hell

#96

Post by throwaway5752 »

coldtea wrote: post #90
pytyper2 wrote: post #53

I'm not sure why the scientists don't use VMs and simply save the virtual disk files? That would at the very least allow them to verify the settings at a later date. Fresh install reproducibility doesn't seem necessary to verify experimental findings as long as the original vm is available to boot up.

VMs are an easy copout to a problem that shouldn't be a problem in the first place.

That's not true. Python has C libraries, some might need to be built from source, and there's good reason to not allow root access on a lot of systems (and ability to install headers/dev packages, gcc, etc). System package management is hard and coordinating with (ubiqitous, not specific to Python) language package managers magnifies it. Unless you had some other solution in mind that I've missed...

Re: Freezing Python’s Dependency Hell

#98

Post by ris »

Here we go again. The source of the problems in in toy package managers (and I include all language package managers here) is not just the package managers themselves, it's the "version soup" philosophy they present to the user. Not daring to risk displeasing the user, they will take orders akin to "I'd like version 1.2.3 of package a, version 31.4.1q of package b, version 0.271 of package c, version 141 of package d...", barely giving a thought to inter-version dependencies of the result.

Unfortunately, software does not work this way. You cannot just ask for an arbitrary combination of versions and rely on it to work. Conflicts and diamond dependencies lurk everywhere.

Sensible package systems (see specifically Nix & nixpkgs) have realized this and follow a "distribution" model where they periodically settle upon a collection of versions of packages which generally are known to work pretty well together (nixpkgs in particular tries to ensure packages' test suites pass in any environment they're going to be installed in). A responsible package distribution will also take it upon themselves to maintain these versions with (often backported) security fixes so that it's no worry sticking with a selection of versions for ~6 months.

However, I can't say I'm particularly surprised that these systems tend to lose out in popularity to the seductively "easy" systems that try to promise the user the moon.

Re: Freezing Python’s Dependency Hell

#99

Post by kalefranz »

jrochkind1 wrote: post #72

ruby practices based around bundler aren't perfect, but they did solve _this_ level of problem ~7 years ago. It remains a mystery to me why python seems to have won the popularity battle against ruby. They are very similar languages, but in all ways they differ ruby seems superior to me.

My theory is that it's because Travis Oliphant wrote numpy for python rather than ruby.

Re: Freezing Python’s Dependency Hell

#100

Post by ben509 »

iooi wrote: post #67
abhishekjha wrote: post #6

What's wrong with pipenv? I am genuinely curious. On local : mkdir my_project_directory cd my_project_directory export PIPENV_VENV_IN_PROJECT=1 (To make the virtual environment folder determininstic(.venv/) otherwise you will get a hash based directory(my_project_directory-some-hash-value) which might not be suitable for automatic deployments in applications like docker. I don't know why this is not default.) pipenv…

pipenv has a huge issue that they refuse to fix: no init command. That means you can only run pipenv commands from the root directory of your project. If you accidentally run pipenv install X in a subdirectory, guess what? You just created a new Pipfile and virtualenv! npm actually got this right, init helps, and it makes sense to traverse up directories to find a package.json.

I agree searching up would be helpful, though, honestly, build tools and other engineers assume you're doing build actions in the project root, and as some of them fail mysteriously if you're not, I often write scripts to fail fast if they're run elsewhere.

Regarding "init", are you complaining that many commands will create a new virtualenv when really only one ought to? Automagically creating the virtualenv definitely seemed cool and modern to me... for about 3 minutes.

Post reply on HN