Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Freezing Python’s Dependency Hell

tech.instacart.com

Post reply on HN
Print view View original

51–60 of 152 posts

Re: Freezing Python’s Dependency Hell

#51

Post by pcl »

syoc wrote: post #36

Why would you do this? Redirect chain: https://tech.instacart.com/freezing-pythons-dependency-hell-in-2018-f1076d625241 https://medium.com/m/global-identity?redirectUrl=https%3A%2F%2Ftech.instacart.com%2Ffreezing-pythons-dependency-hell-in-2018-f1076d625241 https://tech.instacart.com/freezing-pythons-dependency-hell-in-2018-f1076d625241?gi=85c0588ca374

It looks like tech.instacart.com is hosted on Medium. The redirect is part of the auth flow. If you have a Medium account, you would have logged in to medium.com, not tech.instacart.com. If you don't have a Medium account, Medium still will want to add first-party tracking information to your interaction with tech.instacart.com and all other Medium properties. So this client-side redirect flow enables them to capture that association.

This is presumably what the `gi=85c0588ca374` query parameter is in the follow-on redirect. I would guess that `gi` stands for "global identity" or something.

Re: Freezing Python’s Dependency Hell

#52

Post by rb808 »

sandGorgon wrote: post #20

genuine question - is nobody using anaconda/conda in production ? I have found the binary install experience in conda far more pleasant than in anything else. Going forward, the trend is going to be pipenv+manylinux ( https://github.com/pypa/manylinux ), but conda is super pleasant today

Agreed its the most practical solution for most people. Its also a shame that it shows just how unreliable python packaging is.

Re: Freezing Python’s Dependency Hell

#53

Post by pytyper2 »

I'm not sure why the scientists don't use VMs and simply save the virtual disk files? That would at the very least allow them to verify the settings at a later date. Fresh install reproducibility doesn't seem necessary to verify experimental findings as long as the original vm is available to boot up.

Re: Freezing Python’s Dependency Hell

#54

Post by web007 »

Waterluvian wrote: post #9

I ran into a migraine last week: cleaning up requirements.txt How do you determine which requirements are no longer needed when you remove one from your code? In node, your package.json lists only packages YOU installed. So removing them cleans up their dependencies. But in Python, adding one package with pip install might add a dozen entries, none indicating they're dependencies of other packages.

It really bothers me that they're skipping these two as separate steps. Track "what I asked for", use "what I ended up with" for deployment. Otherwise you're just saying "use pip freeze" regardless of wrapping magic around it.

If you're already down that road, pipdeptree is your friend. It will resolve your frozen packages to at least tell you which are top-level and which are dependencies-of-dependencies. There are still exceptions if you're using a dependency both directly and via another module, but having a requirements.in from the pipdeptree parents will have you covered.

Get that list, set them all to module>=version in development, pip install -r requirements.in, then pip freeze > requirements.txt to get hard version locks for deployment.

As others have stated, pip-tools handles this separation for you.

Re: Freezing Python’s Dependency Hell

#55

Post by crdoconnor »

spapas82 wrote: post #45
crdoconnor wrote: post #39

Earlier quoted context omitted.

I hate the whole idea of activating virtualenvs. It's a tool that makes it really easy to end up running a command in the wrong environment and see weird behavior instead of a clear error message. I've seen variations on this scenario happen at least 3 times, for instance: 1) Somebody creates script that activates and runs django and commits it. 2) Junior runs script but the virtualenv doesn't get created for some re…

That's 7 characters more you'll need to write all the time! Also you'll need to remember to prepend them to all scrips, ie pip, fab etc. Well that seems to me to be more error prone for juniors than telling them to always use a virtual env (ie have (envnane) in their prompt)!!

It's less error prone. Never had ^^ that scenario since and I've not run into additional problems either.

Having an extra 7 characters in a ./run.sh script doesn't really bother me. I'm not a perl developer.

Re: Freezing Python’s Dependency Hell

#56

Post by jsmeaton »

crdoconnor wrote: post #42
jsmeaton wrote: post #35

Earlier quoted context omitted.

It really serves a different audience/purpose. Poetry replaces setup.py which is mostly used for building libraries. You still need to create your own virtualenv. Pipenv replaces requirements.txt and handles the virtualenv for you. It can’t be used for packaging libs, but it’s primary purpose is to make developing on apps easier. Lore seems to be much closer to pipenv than poetry.

pyproject.toml and pyproject.lock are analogous to Pipfile and Pipfile.lock Both attempt to replace requirements.txt, both have dependency resolvers and both are workflow tools.

Yet poetry does not manage the virtualenv for you which is what these other tools do. I think poetry would find a lot more love outside of packaging where people are now using pipenv if it managed the virtualenv too.

The integration pipenv has with pyenv is also very nice.

Many people want less tools, which is the primary reason pipenv took off IMO. Creating and activating virtualenvs? mkvirtualenv? Minor python version changes and the venv is toast? Different ways of structuring requirements files? It’s a mess for juniors especially.

People that are packaging libs probably aren’t having the same difficulty with virtualenvs that junior devs are when starting at a company deving on new code bases, learning new processes and tools. But packaging and releasing python libs is challenging, so tooling to help with that is awesome.

A single tool that can do packaging, dependency management, and venv management would be embraced. If poetry doesn’t add venv management, then pipenv should add packaging management.

Re: Freezing Python’s Dependency Hell

#57

Post by devxpy »

abhishekjha wrote: post #6

What's wrong with pipenv? I am genuinely curious. On local : mkdir my_project_directory cd my_project_directory export PIPENV_VENV_IN_PROJECT=1 (To make the virtual environment folder determininstic(.venv/) otherwise you will get a hash based directory(my_project_directory-some-hash-value) which might not be suitable for automatic deployments in applications like docker. I don't know why this is not default.) pipenv…

Pipenv is certainly better than npm. Although, that may be a result of the js ecosystem but still...

Re: Freezing Python’s Dependency Hell

#58

Post by AndrewOMartin »

beagle3 wrote: post #38
AndrewOMartin wrote: post #14

Earlier quoted context omitted.

I was of the same mentality, until yesterday when I watched this PyCon video, uploaded 13 May 2018. https://www.youtube.com/watch?v=GBQAKldqgZs To cut a long story short, if you're happy with virtualenv and pip then that's great, but the idea of pipenv is to replace virtualenv and pip, which means you'll actually have one tool fewer. :)

Genuine question: does pipenv do anything that [mini]conda doesn't?

Everything you need to know about pipenv is in the linked talk. Sorry, I don't know anything about [mini]conda.

Re: Freezing Python’s Dependency Hell

#59

Post by crdoconnor »

jsmeaton wrote: post #56
crdoconnor wrote: post #42

Earlier quoted context omitted.

pyproject.toml and pyproject.lock are analogous to Pipfile and Pipfile.lock Both attempt to replace requirements.txt, both have dependency resolvers and both are workflow tools.

Yet poetry does not manage the virtualenv for you which is what these other tools do. I think poetry would find a lot more love outside of packaging where people are now using pipenv if it managed the virtualenv too. The integration pipenv has with pyenv is also very nice. Many people want less tools, which is the primary reason pipenv took off IMO. Creating and activating virtualenvs? mkvirtualenv? Minor python vers…

>Yet poetry does not manage the virtualenv for you which is what these other tools do.

That's not correct:

https://poetry.eustace.io/docs/basic-usage/#poetry-and-virtu...

>A single tool that can do packaging, dependency management, and venv management would be embraced.

It does all of those.

Re: Freezing Python’s Dependency Hell

#60

Post by jsmeaton »

crdoconnor wrote: post #59
jsmeaton wrote: post #56

Earlier quoted context omitted.

Yet poetry does not manage the virtualenv for you which is what these other tools do. I think poetry would find a lot more love outside of packaging where people are now using pipenv if it managed the virtualenv too. The integration pipenv has with pyenv is also very nice. Many people want less tools, which is the primary reason pipenv took off IMO. Creating and activating virtualenvs? mkvirtualenv? Minor python vers…

>Yet poetry does not manage the virtualenv for you which is what these other tools do. That's not correct: https://poetry.eustace.io/docs/basic-usage/#poetry-and-virtu... >A single tool that can do packaging, dependency management, and venv management would be embraced. It does all of those.

My apologies! I checked the wrong place in the docs before commenting, you are right.

Can you direct it to use a python other than the one linked to poetry? I don’t have CLI access at the moment.

Post reply on HN