Use pew, not virtualenvwrapper, for Python virtualenvs • Hacker News Board

Re: Use pew, not virtualenvwrapper, for Python virtualenvs

#51

Post by jdiez17 » Fri, Sep 08, 2017, 10:05 PM UTC

RayDonnelly wrote: post #48

jdiez17 wrote: post #15

Earlier quoted context omitted.

Because conda (and by extension, Anaconda) harms the Python ecosystem with their non-standard package format. If Continuum Analytics cared about making scientific packages like numpy and scipy more accessible, they would make binary wheels for Linux/Windows/OSX with the MKL. There is no technical reason that prevents them from doing this. Instead they lock you in to their package/environment manager and create confus…

Conda is a package manager for more than Python. It manages libraries that are shared between Python and R and other tools. This isn't possible with a purely Python-based solution. It also does envs better than most other solutions since it will prefer hardlinks which allows environmental isolation at a very small cost in terms of disc space. Full disclosure: I work for Anaconda Inc. where I am trying my best to make…

Shared libraries were invented at a time when disk space was very scarce. In my opinion, the fact that we still use dynamic linking today is largely an accident of history. Literally the only advantage over static linking (or distributing any shared libraries along with your application) is that you save some space, at the cost of requiring a complex package management system. Also, a sufficiently smart filesystem can deduplicate this transparently.

And of course, it makes distribution of any application that uses shared libraries a whole lot more difficult.

Disk space is the cheapest and most abundant computing resource in 2017. If you want to make these packages more widely available, just create wheels/whatever the R equivalent is. Good, well understood, and interoperable tooling for these already exists.

Re: Use pew, not virtualenvwrapper, for Python virtualenvs

#52

Post by kalefranz » Fri, Sep 08, 2017, 10:48 PM UTC

jdiez17 wrote: post #15

sivvy wrote: post #4

Why not use conda?

Because conda (and by extension, Anaconda) harms the Python ecosystem with their non-standard package format. If Continuum Analytics cared about making scientific packages like numpy and scipy more accessible, they would make binary wheels for Linux/Windows/OSX with the MKL. There is no technical reason that prevents them from doing this. Instead they lock you in to their package/environment manager and create confus…

MKL is developed, owned, and licensed by Intel, not Anaconda, Inc. Anaconda distributes it with permission from Intel. If there were to be wheels built and uploaded to PyPI, it would be by Intel. Instead, Intel has chosen conda as the tool they themselves use to deliver their Intel Distribution for Python.

Re: Use pew, not virtualenvwrapper, for Python virtualenvs

#53

Post by jdiez17 » Fri, Sep 08, 2017, 10:52 PM UTC

kalefranz wrote: post #52

jdiez17 wrote: post #15

Earlier quoted context omitted.

Because conda (and by extension, Anaconda) harms the Python ecosystem with their non-standard package format. If Continuum Analytics cared about making scientific packages like numpy and scipy more accessible, they would make binary wheels for Linux/Windows/OSX with the MKL. There is no technical reason that prevents them from doing this. Instead they lock you in to their package/environment manager and create confus…

MKL is developed, owned, and licensed by Intel, not Anaconda, Inc. Anaconda distributes it with permission from Intel. If there were to be wheels built and uploaded to PyPI, it would be by Intel. Instead, Intel has chosen conda as the tool they themselves use to deliver their Intel Distribution for Python.

Anaconda could easily host their own pip index if they wanted. You only need a web server with `autoindex on`.

Re: Use pew, not virtualenvwrapper, for Python virtualenvs

#54

Post by kalefranz » Fri, Sep 08, 2017, 10:55 PM UTC

jdiez17 wrote: post #51

RayDonnelly wrote: post #48

Earlier quoted context omitted.

Conda is a package manager for more than Python. It manages libraries that are shared between Python and R and other tools. This isn't possible with a purely Python-based solution. It also does envs better than most other solutions since it will prefer hardlinks which allows environmental isolation at a very small cost in terms of disc space. Full disclosure: I work for Anaconda Inc. where I am trying my best to make…

Shared libraries were invented at a time when disk space was very scarce. In my opinion, the fact that we still use dynamic linking today is largely an accident of history. Literally the only advantage over static linking (or distributing any shared libraries along with your application) is that you save some space, at the cost of requiring a complex package management system. Also, a sufficiently smart filesystem ca…

Dynamic linking is hugely helpful when, for example, you want to update to the latest openssl without updating half the binaries on your system. That packages are statically including openssl in wheels, and then wheel versions being explicitly pinned in projects, is introducing some juicy attack vectors.

Re: Use pew, not virtualenvwrapper, for Python virtualenvs

#55

Post by kalefranz » Fri, Sep 08, 2017, 11:22 PM UTC

jdiez17 wrote: post #53

kalefranz wrote: post #52

Earlier quoted context omitted.

MKL is developed, owned, and licensed by Intel, not Anaconda, Inc. Anaconda distributes it with permission from Intel. If there were to be wheels built and uploaded to PyPI, it would be by Intel. Instead, Intel has chosen conda as the tool they themselves use to deliver their Intel Distribution for Python.

Anaconda could easily host their own pip index if they wanted. You only need a web server with `autoindex on`.

Would you also propose a `pip install python`?

Re: Use pew, not virtualenvwrapper, for Python virtualenvs

#56

Post by je42 » Sat, Sep 09, 2017, 6:28 PM UTC

gjjrfcbugxbhf wrote: post #25

doozy wrote: post #11

The preferred environment for Python development and deployment changes more than the latest trend in JavaScript frameworks.

Nope it's been virtualenv for at least 3 years.

it is not really. now it is python3.X -m venv which is still virtualenv but again a different implementation with different bugs

Re: Use pew, not virtualenvwrapper, for Python virtualenvs

#57

Post by wink » Sat, Sep 09, 2017, 6:50 PM UTC

abhirag wrote: post #45

smegel wrote: post #43

Earlier quoted context omitted.

We are talking about packaging tools now?

Kindly don't shoot the messenger(i.e. me) but pipenv does the work of pip + virtualenv + requirements.txt so packaging tool seems like a good name for it. Virtual environments aren't a lot of fun without local packages and reproducibility I assume.

Last time I tried pipenv (a month ago?) there was a breaking unresolved bug on Ubuntu 16.04.

I'm not easy to hold grudges, but such a critical low-level tool manages to break for what I see as one of the most likely server environments to run on... I hold off on converting the (annoying, but working) virtualenv(wrapper) stuff. Just not worth it, now that I spent weeks of my life handling the annoyance that is python packaging and deployment.