Overview of Python dependency management tools • Hacker News Board

Re: Overview of Python dependency management tools

#181

Post by dalke » Wed, Jun 03, 2020, 6:30 PM UTC

oblio wrote: post #178

dalke wrote: post #177

Earlier quoted context omitted.

Python's first posting to Usenet (v 0.9) was in 1991. The 1.0 Misc/ACKS from 1994 includes 50 or so external contributions to that point, showing that "1.0" is a somewhat artificial point. Rust's 1.0 was 2015, which is indeed "20 years after Python was" at 1.0, so how is gen220's comment a rewrite? I started using Python around 1.3, and advocating for its broader use (instead of Perl) by 1997. In 1998 I had a job usi…

Rust is a pretty obscure language, when compared to Java, C#, C/C++, Javascript, Python, etc. Rust is a lot better known because of internet fame, which didn't really exist to this magnitude until well after the dotcom crash. My point is, Python's popularity took off in recent years, basically the last 10 years: https://insights.stackoverflow.com/trends?tags=python%2Cjava , primarily due to data science, machine lear…

Where is gen220 rewriting Python history?

Rust is a pretty obscure language now in pretty much the same way that Python was an obscure language then.

Of course the world of programmers was smaller in the 1990s. But if your baseline is the entire world, then probably every programming language outside of Basic, C/C++, and Pascal was obscure in the 1990s. Just like Rust is now.

It feels very much like you have shifted baselines to determine what "obscure" means.

From my view, Python's popularity took off around 2000. That's when I no longer had to tell people what Python was, and when people in my field (cheminformatics) started shifting new code development from Perl to Python. It's also about when I co-founded the Biopython project for bioinformatics. And SWIG in the mid-1990s included Python support because Python was being used to steer supercomputing calculations at LANL.

So your statement that Python's popularity and use in science in general started only in 2010 sounds like revisionism which distorts the actual history with an artificial baseline.

You wrote "with a few web frameworks which were not that popular".

Ummm.... what? Zope was quite popular. The 2001 Python conference had its own Zope track, and the 2002 conferences felt like it was 50% Zope programmers.

Quoting its Wikipedia entry, "Zope has been called a Python killer app, an application that helped put Python in the spotlight". One of the citations is from 2000, at https://web.archive.org/web/20000302033606/http://www.byte.c... , with "there's no killer app that leads people to Perl in the same way that Zope leads people to Python."

Re: Overview of Python dependency management tools

#182

Post by RMPR » Wed, Jun 03, 2020, 6:36 PM UTC

_ZeD_ wrote: post #163

RMPR wrote: post #39

It's worth noting that on Linux it's slightly different because most of the popular libraries can be installed with the system package manager (no problem of dependency management, updates, ...), I rely on alternative solutions only when I want to use a version of a library different from the one shipped with the package manager (which is not that frequent with fast paced distros like Fedora) or when the library is n…

No. Don't mess with os packages and your dev setup. On a very small scale I can adapt to use the os package version. But when you start to work on 4, 5, 15 projects, each of them that need to work with some specific version of some package, you need to detach from the os

And my point is, use the system packages whenever you can, they are there for a reason, while developing atbswp[0], I faced a situation with wxPython, where the only package available on Linux was the one provided by the package manager, it's a known situation[1], the workaround I used was, instead of "detaching from the os" to work with the OS, more specifically, install the package with the OS, then copy the wx folder from the system's site-package to the venv's site-package[2].

0: https://github.com/rmpr/atbswp 1: https://wxpython.org/pages/downloads/ 2: https://github.com/RMPR/atbswp/blob/master/Makefile

Re: Overview of Python dependency management tools

#183

Post by WD-42 » Fri, Jun 05, 2020, 7:24 PM UTC

darkarmani wrote: post #175

WD-42 wrote: post #173

Earlier quoted context omitted.

It's way more than that. Firstly, most Anaconda installations come shipped with libraries like Matplot, numpy, etc. So a lot of people that use conda write software that assumes those libraries are always available e.g leaving them out of requirements.txt or setup.py. Then there's the issue of Anaconda using it's own package repos, so even if you do manage to figure out what packages an Anaconda developed piece of so…

> So a lot of people that use conda write software that assumes those libraries are always available e.g leaving them out of requirements.txt or setup.py. How is that any different than using python.org python? You'd still be unaware of what versions to use. > you're getting a subtlety or maybe not so subtlety different version of it using standard pip, which creates the worst kind of hard to trace bugs. That's way m…

> How is that any different than using python.org python? You'd still be unaware of what versions to use.

Because python.org doesn't ship with numpy, matplotlib, or any of those other packages. Anaconda does, which makes it possible to import those libraries in projects without explicitly listing them as dependencies.

> That's way more of a problem with pip. You have no idea what versions a pip package is pulling in until install and then what binary actually gets installed depends on your compilers.

What? The problem here is that conda has it's own repos, which contains different packages than are contained in PyPi. What exactly do you mean by "no idea what versions a pip package is pulling". You realize you can set versions, right? numpy==1.13.2. The problem is numpy 1.13 on Anaconda can be different than numpy 1.13 on PyPi.

> That's impossible unless one is actually copying binaries manually overtop of system binaries. You'd have to be root or use sudo to overwrite the system python manually. The whole point of isolation is to keep system python isolated and stable for system stability. That can happen if someone installs python from python.org and copies it into place.

This is just wrong. Anaconda overwrites the system python by messing with the user's $PATH regardless if you are in a conda environment or not (probably easy to disable this "feature" but I've seen a lot of people with this setup). This causes major headaches.

> If you are working with SW engineers, you better know what versions you are pulling in, because you are going to be in serious pain using pip and trying to understand the provenance of your packages. Conda is way more powerful here for serious engineers to specify exact versions and reproducible and exact builds.

I'm not sure why you think you can't specify exact versions with pip. Projects like pipfile take it even further. The issue with conda is it's different package repos, not the ability to lock package versions.

Re: Overview of Python dependency management tools

#184

Post by darkarmani » Fri, Jun 05, 2020, 9:36 PM UTC

WD-42 wrote: post #183

darkarmani wrote: post #175

Earlier quoted context omitted.

> So a lot of people that use conda write software that assumes those libraries are always available e.g leaving them out of requirements.txt or setup.py. How is that any different than using python.org python? You'd still be unaware of what versions to use. > you're getting a subtlety or maybe not so subtlety different version of it using standard pip, which creates the worst kind of hard to trace bugs. That's way m…

> How is that any different than using python.org python? You'd still be unaware of what versions to use. Because python.org doesn't ship with numpy, matplotlib, or any of those other packages. Anaconda does, which makes it possible to import those libraries in projects without explicitly listing them as dependencies. > That's way more of a problem with pip. You have no idea what versions a pip package is pulling in…

> Anaconda does, which makes it possible to import those libraries in projects without explicitly listing them as dependencies.

I think your main problems are very naive users of conda. If you bring years of experience using pip, but use conda thoughtlessly, I can see your point.

If you don't want packages included, just use miniconda and install the ones you like. You could just create a new empty environment: `conda create -n py36 python=3.6`

Either way, it's completely reproducible.

When not using wheels, pip can be pulling in various versions of dependencies. Conda makes it easy to see all of them before they are dumped into your environment.

> Anaconda overwrites the system python by messing with the user's $PATH regardless

I understand what you are saying now. It's covering up system python in the PATH, but it isn't overwritten. Using `type python` (or which python will be correct 99% of the time).

> The issue with conda is it's different package repos, not the ability to lock package versions.

I thought this was your major argument. "Collaboration is difficult" when in fact it is much, much easier. You are getting the same binary everytime without slight differences in how it ends up compiled on the user's system.