Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Freezing Python’s Dependency Hell

tech.instacart.com

Post reply on HN
Print view View original

111–120 of 152 posts

Re: Freezing Python’s Dependency Hell

#111

Post by mos_basik »

iooi wrote: post #67
abhishekjha wrote: post #6

What's wrong with pipenv? I am genuinely curious. On local : mkdir my_project_directory cd my_project_directory export PIPENV_VENV_IN_PROJECT=1 (To make the virtual environment folder determininstic(.venv/) otherwise you will get a hash based directory(my_project_directory-some-hash-value) which might not be suitable for automatic deployments in applications like docker. I don't know why this is not default.) pipenv…

pipenv has a huge issue that they refuse to fix: no init command. That means you can only run pipenv commands from the root directory of your project. If you accidentally run pipenv install X in a subdirectory, guess what? You just created a new Pipfile and virtualenv! npm actually got this right, init helps, and it makes sense to traverse up directories to find a package.json.

They may have changed that behavior recently. I was trying out pipenv last week and running `pipenv run script.py` in a subdirectory printed a message along the lines of "Courtesy notice: no pipfile found in this directory. Using pipfile in [project_root]. Behavior can be customized by specifying a pipfile with [some_flag]". I'm fairly sure, but not positive, that I also was able to install modules from subdirectories into the project venv like you want.

On mobile; I may be misremembering some details. Would encourage you to check new version behavior if you're interested.

Re: Freezing Python’s Dependency Hell

#113

Post by mcintyre1994 »

tamatsyk wrote: post #63

Interesting reading, I share some of the points in the post, however, one more dependency manager? Mostly I've used plain `python -m venv venv` and it always worked well. A downside - you need to add a few bash scripts to automate typical workflow for your teammates. Pipenv sounds great but there are some pitfalls as well. I've been going through this post recently and got a bit upset about Pipenv: https://chriswarri…

> Another point is that it does not work well with PyCharm and does not allow to put all dependencies into the project folder as I used to do with venv.

This is annoying for AWS lambdas too, because you have to bundle the dependencies and zip it. It's pretty trivial to go Pipfile -> requirements.txt -> pip install -t if you use a Makefile, but it's definitely an omission. I asked about it on their github though and it is a known issue, hopefully it'll be there soon.

Re: Freezing Python’s Dependency Hell

#114

Post by rbanffy »

Version pinning is technical debt and a fool's errand. New versions will always come out and your new development is confined to what once worked. You need to keep testing with current versions to see what will break when you upgrade and fix it as soon as possible so as to minimize the odds of a big breaking change.

It may keep your environment stable for some time, but that stability is an illusion because the whole world moves on. You may be able to still keep your Python 2.2 applications running on Centos 3 forever, but you shouldn't want to do it.

Re: Freezing Python’s Dependency Hell

#115

Post by kenhwang »

kalefranz wrote: post #99
jrochkind1 wrote: post #72

ruby practices based around bundler aren't perfect, but they did solve _this_ level of problem ~7 years ago. It remains a mystery to me why python seems to have won the popularity battle against ruby. They are very similar languages, but in all ways they differ ruby seems superior to me.

My theory is that it's because Travis Oliphant wrote numpy for python rather than ruby.

And Python is taught in the intro to programming course in just about every college in the world.

Dumb simple languages make better teaching tools, but unlike Lisp and Smalltalk, Python was also good enough for widespread professional use.

So almost everyone is exposed to Python, many people never bothered to learn anything better. Inertia is a hell of a force.

Re: Freezing Python’s Dependency Hell

#116

Post by tamatsyk »

mcintyre1994 wrote: post #113
tamatsyk wrote: post #63

Interesting reading, I share some of the points in the post, however, one more dependency manager? Mostly I've used plain `python -m venv venv` and it always worked well. A downside - you need to add a few bash scripts to automate typical workflow for your teammates. Pipenv sounds great but there are some pitfalls as well. I've been going through this post recently and got a bit upset about Pipenv: https://chriswarri…

> Another point is that it does not work well with PyCharm and does not allow to put all dependencies into the project folder as I used to do with venv. This is annoying for AWS lambdas too, because you have to bundle the dependencies and zip it. It's pretty trivial to go Pipfile -> requirements.txt -> pip install -t if you use a Makefile, but it's definitely an omission. I asked about it on their github though and i…

JetBrains have heard the prayers :D Here is an announce of pipenv support: https://blog.jetbrains.com/pycharm/2018/06/pycharm-2018-2-ea...

> because you have to bundle the dependencies and zip it btw, I've used serverless to deploy lambdas in python and it worked super cool. Highly recommended.

Re: Freezing Python’s Dependency Hell

#117

Post by tamatsyk »

peterwwillis wrote: post #69
tamatsyk wrote: post #63

Interesting reading, I share some of the points in the post, however, one more dependency manager? Mostly I've used plain `python -m venv venv` and it always worked well. A downside - you need to add a few bash scripts to automate typical workflow for your teammates. Pipenv sounds great but there are some pitfalls as well. I've been going through this post recently and got a bit upset about Pipenv: https://chriswarri…

Actually, I recommend bash scripts for automating team workflows as a best practice. You create a wrapper script around your application that calls a dev environment set-up script, that [if it wasn't done yet] sets up the environment from scratch for that project or application, and loads it before running your application. This does a couple things. First, it removes the need to train anyone on using your best pract…

Completely agree on every bullet point,

Every time I saw simple bash scripts or/and Makefile used - it did not seem to be the idiomatic way of doing things in python but after using it for a while - turned out to be one of the best development experiences.

Re: Freezing Python’s Dependency Hell

#118

Post by mkobit »

kalefranz wrote: post #88
mkobit wrote: post #68

Earlier quoted context omitted.

Not OP, but we would like to move away from it as well. - Breaking behavior between minor versions ( https://github.com/conda/conda/issues/7290 ) - Environments not actually being isolated ( https://github.com/conda/conda/issues/448 ) - Can't create environments in long paths ( https://github.com/conda/constructor/issues/156 ) Those are just a few I can remember. We unfortunately have not found a strong replacement.

> Breaking behavior between minor versions See https://github.com/conda/conda/issues/7248 for where conda intends to head in the future on the environment.yml issue. > Environments not actually being isolated That's actually a really sticky issue, and one that's more about the python interpreter itself rather than anything conda is doing. More recent discussion at https://github.com/conda/conda/issues/7173 . Yes, we…

Thanks for the reply. I wrote that post earlier this morning after some other frustration, so I'm sorry for the negative tone.

Despite some of our issues with it, Conda has worked well for us both as a development team and in production for quite some time.

> six million active users

That's awesome to hear, and one of the reasons we haven't seriously invested in a replacement.

Re: Freezing Python’s Dependency Hell

#119

Post by scroot »

kenhwang wrote: post #115
kalefranz wrote: post #99

Earlier quoted context omitted.

My theory is that it's because Travis Oliphant wrote numpy for python rather than ruby.

And Python is taught in the intro to programming course in just about every college in the world. Dumb simple languages make better teaching tools, but unlike Lisp and Smalltalk, Python was also good enough for widespread professional use. So almost everyone is exposed to Python, many people never bothered to learn anything better. Inertia is a hell of a force.

When MIT stopped teaching Scheme (replaced with Python) a couple of years back, yet another essential concept in computing left the academy. This is exactly the kind of thing Kay means when he talks about computing pop-culture.

Anyone who's ever read The Little Schemer knows what I mean.

Re: Freezing Python’s Dependency Hell

#120

Post by kalefranz »

cuchoi wrote: post #112
superbatfish wrote: post #107

The author's justifications for using this home-grown tool over miniconda are weak at best, if not plain incorrect. Conda really is the tool he wants; he just seems not to understand that.

How does Conda replaces a virtual environment? (honest question)

Python's virtualenvs target isolation of the site-packages directory. Conda environments are one step up in abstraction, isolating the "prefix" (in python world just the output of `sys.prefix`). The target for conda and conda environments is the management of everything within that prefix, including python itself. The target for pip, pipenv, virtualenv, and other python-only package management tools is everything within `lib/pythonX.Y/site-packages`.

The distinction is important especially for people using python's data science libraries, since those libraries are often just python wrappers around compiled code and link to shared "system" libraries. Conda manages and isolates those libraries; pip and virtualenv do not.

The distinction also has security implications, for example when openssl is statically embedded in wheels. When this happens, there isn't any real visibility into the openssl versions being used. Because conda has the flexibility of the step up in abstraction as I described before, conda can manage a single instance of openssl for the whole environment, and then the python packages needing openssl need not statically embed it.

Post reply on HN