Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Freezing Python’s Dependency Hell

tech.instacart.com

Post reply on HN
Print view View original

1–10 of 152 posts

Re: Freezing Python’s Dependency Hell

#2

Post by michaelmcmillan »

Never had a problem with dependencies in Python. Just keep it simple.

When starting a new project:

  virtualenv venv -p *path-to-python-version-you-want*
  ./venv/bin/pip install *name-of-package*
When running that project:

  ./venv/bin/python *name-of-python-file*
Many people don't realize that the venv/bin/ contains all the relevant binaries with the right library path's out of the box.

Re: Freezing Python’s Dependency Hell

#3

Post by Rjevski »

michaelmcmillan wrote: post #2

Never had a problem with dependencies in Python. Just keep it simple. When starting a new project: virtualenv venv -p *path-to-python-version-you-want* ./venv/bin/pip install *name-of-package* When running that project: ./venv/bin/python *name-of-python-file* Many people don't realize that the venv/bin/ contains all the relevant binaries with the right library path's out of the box.

Pipenv is a good replacement for the above workflow. It manages your dependencies and virtualenvs.

Re: Freezing Python’s Dependency Hell

#4

Post by zedr »

Using a local virtual environment and then building a Docker image removes most of the headaches. I also bundle a Makefile with simple targets. See this as an example: https://github.com/zedr/cffi_test/blob/master/Makefile New projects are created from a template using Cookiecutter.

It isn't really so bad in 2018, but I do have a lot of scars from the old days, most of them caused by zc.buildout.

The secret is using, as the article mentions, a custom virtual env for each instance of the project. I never found the need for stateful tooling like Virtualenvwrapper.

Re: Freezing Python’s Dependency Hell

#5

Post by aequitas »

"Pipfile looks promising for managing package dependencies, but is under active development. We may adopt this as an alternative if/when it reaches maturity, but for the time being we use requirements.txt."

If I where given the choice between community supported/in development Pipfile/pipenv or the 3rd party supported yet-another-package-manager lore to get those best practices my money would be on Pipfile/pipenv. I've been using it for many project now and besides some minor annoyances (eg: the maintainer's love for color output that is not form follow function) it has been a great tool.

Re: Freezing Python’s Dependency Hell

#6

Post by abhishekjha »

What's wrong with pipenv? I am genuinely curious.

On local :

    mkdir my_project_directory
    cd my_project_directory
    export PIPENV_VENV_IN_PROJECT=1 (To make the virtual environment folder determininstic(.venv/) otherwise you will get a hash based directory(my_project_directory-some-hash-value) which might not be suitable for automatic deployments in applications like docker. I don't know why this is not default.)
    pipenv --python 3.6 (or any particular version number)
    pipenv install numpy scipy pandas matplotlib requests
    pipenv graph (Gives me a dependency graph)
    git add .
    git commit -a -S -m "init" 
    git push
On remote :

    git clone url/my_project_directory
    cd my_project_directory
    export PIPENV_VENV_IN_PROJECT=1
    pipenv install
    pipenv shell
    pipenv graph
Is this workflow not enough? I have recently started using pipenv after a lot of struggle. The only issue I have is, Pycharm doesn't allow native pipenv initialisation. I always end up creating an environment manually and then importing the project. Pycharm does detect the environment though.

Re: Freezing Python’s Dependency Hell

#7

Post by michaelmcmillan »

Rjevski wrote: post #3
michaelmcmillan wrote: post #2

Never had a problem with dependencies in Python. Just keep it simple. When starting a new project: virtualenv venv -p *path-to-python-version-you-want* ./venv/bin/pip install *name-of-package* When running that project: ./venv/bin/python *name-of-python-file* Many people don't realize that the venv/bin/ contains all the relevant binaries with the right library path's out of the box.

Pipenv is a good replacement for the above workflow. It manages your dependencies and virtualenvs.

Thanks for the tip, but honestly, I don't need another tool.

Re: Freezing Python’s Dependency Hell

#8

Post by ergo14 »

abhishekjha wrote: post #6

What's wrong with pipenv? I am genuinely curious. On local : mkdir my_project_directory cd my_project_directory export PIPENV_VENV_IN_PROJECT=1 (To make the virtual environment folder determininstic(.venv/) otherwise you will get a hash based directory(my_project_directory-some-hash-value) which might not be suitable for automatic deployments in applications like docker. I don't know why this is not default.) pipenv…

Up till not long ago pipenv was not ready - for example it could not install packages like gevent. That bug is fixed now I believe.

Re: Freezing Python’s Dependency Hell

#9

Post by Waterluvian »

I ran into a migraine last week: cleaning up requirements.txt

How do you determine which requirements are no longer needed when you remove one from your code? In node, your package.json lists only packages YOU installed. So removing them cleans up their dependencies. But in Python, adding one package with pip install might add a dozen entries, none indicating they're dependencies of other packages.

Re: Freezing Python’s Dependency Hell

#10

Post by neuland »

Waterluvian wrote: post #9

I ran into a migraine last week: cleaning up requirements.txt How do you determine which requirements are no longer needed when you remove one from your code? In node, your package.json lists only packages YOU installed. So removing them cleans up their dependencies. But in Python, adding one package with pip install might add a dozen entries, none indicating they're dependencies of other packages.

I’m not sure about other people, but that is how I use requirements.txt. You don’t have to dump the entire output of pip freeze in there. You can just list the dependencies you want.
Post reply on HN