Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Freezing Python’s Dependency Hell

tech.instacart.com

Post reply on HN
Print view View original

141–150 of 152 posts

Re: Freezing Python’s Dependency Hell

#141

Post by craigds »

kalefranz wrote: post #131
sandGorgon wrote: post #129

Earlier quoted context omitted.

Quick question - has there been an attempt to make conda and official PEP and make conda-forge part of the Python Foundation (instead of a private company). I'm trying to figure out why is all this new manylinux PEP stuff "inspired by conda" and is not actually conda. Is this a situation like grsecurity vs Torvalds? How does the situation change now that BDFL is gone ?

> Is this a situation like grsecurity vs Torvalds? Ha! Conda and Anaconda, Inc. are _not_ like grsecurity. Starting with the fact the Conda is BSD-licensed. Recounting some of the history will probably provide the context you're looking for. It's my understanding that the birth of Anaconda (the distribution) and Conda goes back to this statement by Guido regarding package building and installing at a PyData summit in…

Binary wheels are a nightmare in terms of knowing what you have installed. We disabled them in our builds when we discovered a few packages had vendored old, terribly-insecure static versions of libxml2 (and who knows what else) which silently replaced using the system version (which received OS security updates). We only found out when we hit one of the behavioural quirks of that version and spent ages investigating it with gdb.

If you're using manylinux wheels you're probably using static libs you have no idea you're using. That way lies madness.

I havent used conda so maybe this doesn't apply there somehow.

Re: Freezing Python’s Dependency Hell

#142

Post by terrelln »

handruin wrote: post #74
dorfsmay wrote: post #31

Anybody played with the brand new XAR from Facebook? https://code.fb.com/data-infrastructure/xars-a-more-efficien...

Thanks for the link. That looks interesting; I'll have to give that a try. When I started reading the link my first thought was Pex from Twitter. I don't know how comparable XAR is to Pex but it's worth a look to compare the two.

Both PEXs and XARs package a python script and its dependencies in single hermetic file.

PEX is a self-extracting zip file which has to be fully extracted before being run. The extracted files could potentially be modified.

XAR is a self-mounting compressed SquashFS filesystem image. SquashFS will decompress pages lazily and cache the result in the page cache, so the startup time is much faster. Since SquashFS is read-only, the files can't be modified.

Re: Freezing Python’s Dependency Hell

#143

Post by mcintyre1994 »

tamatsyk wrote: post #116
mcintyre1994 wrote: post #113

Earlier quoted context omitted.

> Another point is that it does not work well with PyCharm and does not allow to put all dependencies into the project folder as I used to do with venv. This is annoying for AWS lambdas too, because you have to bundle the dependencies and zip it. It's pretty trivial to go Pipfile -> requirements.txt -> pip install -t if you use a Makefile, but it's definitely an omission. I asked about it on their github though and i…

JetBrains have heard the prayers :D Here is an announce of pipenv support: https://blog.jetbrains.com/pycharm/2018/06/pycharm-2018-2-ea... > because you have to bundle the dependencies and zip it btw, I've used serverless to deploy lambdas in python and it worked super cool. Highly recommended.

Oo nice I didn't know serverless worked with Python! Thanks for the heads up :)

Re: Freezing Python’s Dependency Hell

#144

Post by yrro »

_ikke_ wrote: post #17
yrro wrote: post #11

Earlier quoted context omitted.

It uses virtualenv, rather than venv. After discovering PYTHONUSERBASE, I no longer need any of the plethora of wrappers around venv/virtualenv.

How do you use PYTHONUSERBASE?

    $ export PYTHONUSERBASE=/path/to/project-specific-python-user-base
    $ python3 -m pip install --user whatever
Everything now goes into that directory. Different projects on the same machine have different directories and so can't affect one another.

Re: Freezing Python’s Dependency Hell

#145

Post by yrro »

abhishekjha wrote: post #49
yrro wrote: post #11

Earlier quoted context omitted.

It uses virtualenv, rather than venv. After discovering PYTHONUSERBASE, I no longer need any of the plethora of wrappers around venv/virtualenv.

>After discovering PYTHONUSERBASE, I no longer need any of the plethora of wrappers around venv/virtualenv. Is there any walkthrough available? Pipenv has one deficiency is that it can be slow at times, particularly when you want to quickly install a dependency and run. Would love to know the alternative.

https://docs.python.org/3/using/cmdline.html#envvar-PYTHONUS...

Set the environment variable to a different directory for each project, and `python3 -m pip install whatever` will go into that directory.

Re: Freezing Python’s Dependency Hell

#146

Post by jrochkind1 »

kenhwang wrote: post #115
kalefranz wrote: post #99

Earlier quoted context omitted.

My theory is that it's because Travis Oliphant wrote numpy for python rather than ruby.

And Python is taught in the intro to programming course in just about every college in the world. Dumb simple languages make better teaching tools, but unlike Lisp and Smalltalk, Python was also good enough for widespread professional use. So almost everyone is exposed to Python, many people never bothered to learn anything better. Inertia is a hell of a force.

> And Python is taught in the intro to programming course in just about every college in the world.

Why do you think that ended up python instead of ruby? Something about the language or it's uses, or just a coincidence of history?

I have no idea myself.

I think ruby and python are about equal level of both "simpleness" (neither is very simple, actually; although it depends on what you mean by 'simple') and "good enough for widespread professional use" (both are, and especially both were ~8 years ago). Or do you disagree and think they differ there?

Re: Freezing Python’s Dependency Hell

#147

Post by kenhwang »

jrochkind1 wrote: post #146
kenhwang wrote: post #115

Earlier quoted context omitted.

And Python is taught in the intro to programming course in just about every college in the world. Dumb simple languages make better teaching tools, but unlike Lisp and Smalltalk, Python was also good enough for widespread professional use. So almost everyone is exposed to Python, many people never bothered to learn anything better. Inertia is a hell of a force.

> And Python is taught in the intro to programming course in just about every college in the world. Why do you think that ended up python instead of ruby? Something about the language or it's uses, or just a coincidence of history? I have no idea myself. I think ruby and python are about equal level of both "simpleness" (neither is very simple, actually; although it depends on what you mean by 'simple') and "good eno…

I love Ruby, but I would still advocate for Python as a teaching language.

Ruby's grammar is objectively more complex than Python's. People generally get stuck on syntax issues when they begin learning programming. Python's significantly simpler grammar, simpler and fewer basic building blocks, and historically "only one way to do things" philosophy makes it easier to pick up.

Ruby's conventional control flow also doesn't translate well to lower level languages; Enumerable pretty much replaces all sorts of loops, functions/methods tend to be chained instead of wrapped (math style), implicit returns, and perlisms, all make Ruby more confusing for a first timer language.

Re: Freezing Python’s Dependency Hell

#148

Post by RayDonnelly »

craigds wrote: post #141
kalefranz wrote: post #131

Earlier quoted context omitted.

> Is this a situation like grsecurity vs Torvalds? Ha! Conda and Anaconda, Inc. are _not_ like grsecurity. Starting with the fact the Conda is BSD-licensed. Recounting some of the history will probably provide the context you're looking for. It's my understanding that the birth of Anaconda (the distribution) and Conda goes back to this statement by Guido regarding package building and installing at a PyData summit in…

Binary wheels are a nightmare in terms of knowing what you have installed. We disabled them in our builds when we discovered a few packages had vendored old, terribly-insecure static versions of libxml2 (and who knows what else) which silently replaced using the system version (which received OS security updates). We only found out when we hit one of the behavioural quirks of that version and spent ages investigating…

We are a rolling Software Distribution and providing up to date software is one of our main goals.

Another is building that software with good security flags, see: https://www.anaconda.com/blog/developer-blog/improved-securi...

We also keep track of CVEs in our software and actively look for patches (e.g. pycrypto is dead now but Debian maintains patches to fix reported CVEs) or write our own (though usually to fix build-system bugs rather than security issues).

But yes, static linking and leaving software building to non-experts using whatever tools they like (without studying anything to do with low-level binary security or how to achieve that), statically linking insecure (some time to become old) libraries is far from ideal.

Anaconda Distribution strongly prefers dynamic linking and shared package dependencies so we can update to address critical security issues without needing to rebuild significant portions of our stack.

Re: Freezing Python’s Dependency Hell

#149

Post by RayDonnelly »

throwaway5752 wrote: post #137
hmaarrfk wrote: post #106

Earlier quoted context omitted.

conda basically proves that you can install almost everything one needs in the user's home directory. they have been working more and more on being completely independent from things like system compilers as well.

Goodness, you say that like it's a good thing. Yes, it is easy to download compiled binaries from a 3rd party.

I don't understand the nature of your discourse. You agreed that maintaining software distros is not easy, some recommended conda and you seem dismissive again?
Post reply on HN