Live data from Hacker News

Freezing Python’s Dependency Hell

tech.instacart.com

131–140 of 152 posts

Re: Freezing Python’s Dependency Hell

#131

Earlier quoted context omitted.

> Breaking behavior between minor versions See https://github.com/conda/conda/issues/7248 for where conda intends to head in the future on the environment.yml issue. > Environments not actually being isolated That's actually a really sticky issue, and one that's more about the python interpreter itself rather than anything conda is doing. More recent discussion at https://github.com/conda/conda/issues/7173 . Yes, we…

Quick question - has there been an attempt to make conda and official PEP and make conda-forge part of the Python Foundation (instead of a private company). I'm trying to figure out why is all this new manylinux PEP stuff "inspired by conda" and is not actually conda. Is this a situation like grsecurity vs Torvalds? How does the situation change now that BDFL is gone ?

> Is this a situation like grsecurity vs Torvalds?

Ha! Conda and Anaconda, Inc. are _not_ like grsecurity. Starting with the fact the Conda is BSD-licensed.

Recounting some of the history will probably provide the context you're looking for.

It's my understanding that the birth of Anaconda (the distribution) and Conda goes back to this statement by Guido regarding package building and installing at a PyData summit in 2012:

> It really sounds like you guys' needs are so unusual compared to the larger python community that you're just better off building your own.

https://www.youtube.com/watch?v=QjXJLVINsSA&t=59m10s

This predated PEP 427 (Wheel Binary Package Format 1.0).

Of course both conda and pip/wheels have evolved immensely since 2012, and their evolution has been guided by different constraints. Elsewhere in this thread I spoke about how pip/wheels/PyPA packaging has as it's primary target the site-packages directory, and then branches out from there when necessary. Conda's primary target is the prefix root. PyPA's mandate is to build, install, and manage python packages, which makes something like 'pip install python3' out of scope. Things like 'pip install postgresql' and 'pip install r' are _for sure_ out of scope. Conda has the luxury of being able to install and manage all packages, not just python packages.

Regarding the creep of wheels toward conda packages, and especially the static embedding within wheels of compiled "system" libraries, I question whether it's actually to the benefit of python _users_. No doubt the wheel "binary" format is a huge improvement over eggs for pure-python packages. But manylinux wheels are often built going beyond just compiled python extensions. Rather than describing these "system" dependencies in metadata, and then having pip ensure these system dependencies are present before installing the package, these wheels now implicitly cross the python-only line that PyPA in other cases holds. There are real consequences for users, with the result being that it pushes failure mechanisms back rather forward.

So, because of the differences in scope, my guess is that the python community would think it inappropriate to make Conda an official PEP. Conda will probably someday install wheels (at least pure-python ones); maybe someday pip can reach out to conda or apt-get or yum to ask for non-python dependencies to be installed.

> and make conda-forge part of the Python Foundation (instead of a private company)

Conda-forge is actually a community-driven organization completely independent of Anaconda, Inc.

Re: Freezing Python’s Dependency Hell

#132
post #124

Earlier quoted context omitted.

There's been quite a bit of discussion about Anaconda and conda in this thread already. Anaconda also takes this distribution approach, and it's targeted specifically at python.

Yet it will never be able to solve the system-library dependency problem in the way that Nix does.

Can you elaborate? Conda packages have full metadata descriptions for system-level dependencies. I agree with you that conflicts and diamond dependencies lurk everywhere, which is precisely the reason Conda employs a SAT solver.

Re: Freezing Python’s Dependency Hell

#133

Earlier quoted context omitted.

Quick question - has there been an attempt to make conda and official PEP and make conda-forge part of the Python Foundation (instead of a private company). I'm trying to figure out why is all this new manylinux PEP stuff "inspired by conda" and is not actually conda. Is this a situation like grsecurity vs Torvalds? How does the situation change now that BDFL is gone ?

> Is this a situation like grsecurity vs Torvalds? Ha! Conda and Anaconda, Inc. are _not_ like grsecurity. Starting with the fact the Conda is BSD-licensed. Recounting some of the history will probably provide the context you're looking for. It's my understanding that the birth of Anaconda (the distribution) and Conda goes back to this statement by Guido regarding package building and installing at a PyData summit in…

> my guess is that the python community would think it inappropriate to make Conda an official PEP.

will you guys try ? I really, really, really hope you do.

Re: Freezing Python’s Dependency Hell

#134

What's wrong with pipenv? I am genuinely curious. On local : mkdir my_project_directory cd my_project_directory export PIPENV_VENV_IN_PROJECT=1 (To make the virtual environment folder determininstic(.venv/) otherwise you will get a hash based directory(my_project_directory-some-hash-value) which might not be suitable for automatic deployments in applications like docker. I don't know why this is not default.) pipenv…

Pipenv is probably the best tool to keep clean the dependency workflow

Re: Freezing Python’s Dependency Hell

#135

I'm not sure why the scientists don't use VMs and simply save the virtual disk files? That would at the very least allow them to verify the settings at a later date. Fresh install reproducibility doesn't seem necessary to verify experimental findings as long as the original vm is available to boot up.

My guesses are that: 1. Integrating the development environment on their host PC (for example connecting RStudio in R's case, or connecting their web browser back to a server running in the VM in the case of Jupyter) is another set of skills to master. 2. Many data analyses are memory hungry unless you want to resort to coding practices that optimize for memory consumption. The overhead of running a VM is a bummer fo…

I think these and other issues can be solved with technical training.

Re: Freezing Python’s Dependency Hell

#136

What's wrong with pipenv? I am genuinely curious. On local : mkdir my_project_directory cd my_project_directory export PIPENV_VENV_IN_PROJECT=1 (To make the virtual environment folder determininstic(.venv/) otherwise you will get a hash based directory(my_project_directory-some-hash-value) which might not be suitable for automatic deployments in applications like docker. I don't know why this is not default.) pipenv…

Pycharm just released 2018.2 with support for pipenv :) https://www.jetbrains.com/pycharm/whatsnew/#v2018-2-python

So I went ahead and tried it. To not have `export PIPENV_VENV_IN_PROJECT=1` while creating an environment is giving me an environment with the hash value. This is not good. Looks like I am gonna have to keep creating pipenv environments manually and then import projects into pycharm.

Re: Freezing Python’s Dependency Hell

#137

Earlier quoted context omitted.

That's not true. Python has C libraries, some might need to be built from source, and there's good reason to not allow root access on a lot of systems (and ability to install headers/dev packages, gcc, etc). System package management is hard and coordinating with (ubiqitous, not specific to Python) language package managers magnifies it. Unless you had some other solution in mind that I've missed...

conda basically proves that you can install almost everything one needs in the user's home directory. they have been working more and more on being completely independent from things like system compilers as well.

Goodness, you say that like it's a good thing. Yes, it is easy to download compiled binaries from a 3rd party.

Re: Freezing Python’s Dependency Hell

#138
post #67

Earlier quoted context omitted.

pipenv has a huge issue that they refuse to fix: no init command. That means you can only run pipenv commands from the root directory of your project. If you accidentally run pipenv install X in a subdirectory, guess what? You just created a new Pipfile and virtualenv! npm actually got this right, init helps, and it makes sense to traverse up directories to find a package.json.

They may have changed that behavior recently. I was trying out pipenv last week and running `pipenv run script.py` in a subdirectory printed a message along the lines of "Courtesy notice: no pipfile found in this directory. Using pipfile in [project_root]. Behavior can be customized by specifying a pipfile with [some_flag]". I'm fairly sure, but not positive, that I also was able to install modules from subdirectorie…

Wow thank you so much! It's crazy how much this project is changing[1], especially and more and more users/companies/projects are starting to depend on it.

[1] https://github.com/pypa/pipenv/blob/77110ed5da89823fa5954e47...

Re: Freezing Python’s Dependency Hell

#139

Earlier quoted context omitted.

My guesses are that: 1. Integrating the development environment on their host PC (for example connecting RStudio in R's case, or connecting their web browser back to a server running in the VM in the case of Jupyter) is another set of skills to master. 2. Many data analyses are memory hungry unless you want to resort to coding practices that optimize for memory consumption. The overhead of running a VM is a bummer fo…

I think these and other issues can be solved with technical training.

Sure, they’re all mitigatable, but that technical training is competing with a lot of other considerations within the limited brainwidth of a scientist.

From the scientist’s perspective, a lot of this can start to feel like yak shaving. The opportunity costs are real.

Re: Freezing Python’s Dependency Hell

#140

The author's justifications for using this home-grown tool over miniconda are weak at best, if not plain incorrect. Conda really is the tool he wants; he just seems not to understand that.

> if not plain incorrect The justification was that the Anaconda installer is too heavy. The kitchen sink Anaconda installer is not designed for the author's use case. Miniconda is the provided way to bootstrap conda onto a system.

Indeed, Anaconda is too heavy. But he is aware of Miniconda and even mentions it in his last bullet point. He then dismisses it in short order, with a vague complaint about mysterious "best practices".

If he really believes that his tool is somehow better, fine. But since Miniconda is the de facto standard tool among data scientists for this use-case, the burden is on him to spend more words on exactly why it doesn't work for him.

Post reply on HN