Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python: Please stop screwing over Linux distros

drewdevault.com

Post reply on HN
Print view View original

371–380 of 384 posts

Re: Python: Please stop screwing over Linux distros

#371

Post by PeterisP »

goodpoint wrote: post #307
goodpoint wrote: post #240

Earlier quoted context omitted.

Go is terrible for distributions to package due to the poor versioning of libraries.

Despite the downvotes, the argument stands: linux distributions are having a hard time handle the amount of tiny libraries and the conflicts in versioning and many maintainers voiced their concerns in the past years. The HN bubble can be amazing sometimes.

The point echoed in this discussion multiple times is that distributions should not handle the tiny python libraries and attempt to solve the dependency version issues, but treat an application with all its dependencies included as a single package. If a dependency needs to be bumped a version for e.g. security purposes, then the app obviously wasn't tested with the new version (which didn't exist at the time) and needs to be retested, repackaged and rereleased for the update. This would cut down on the number of packages to be maintained, as the vast majority of python libraries would be exclude from the direct packaging process.

Re: Python: Please stop screwing over Linux distros

#372

Post by PeterisP »

WhyNotHugo wrote: post #75
cdent wrote: post #46

I'm always confused by these sorts of posts because they happen often so there is clearly a problem but for some reason I've never had much of an issue. I've been using and developing with Python for about 15 years. In that time I've worked on Python projects large (OpenStack) and small (gabbi) and taken over maintenance for some old standbys (wsgi-intercept, paste to name two). Dealt with the 2->3 transition. Releas…

So there's a few types of projects you can write in Python: 1. Server applications that run in a dedicated environment. 2. Tools you write and run just on your machine (or some virtualenv, whatever). 3. Redistributable cli or desktop applications which end users will install and use. For the first two types, you should never have any issues with Python and its dependency situation. You pin everything, and that's it.…

#3 is the exact case where you also want to just pin everything. For an end-user desktop application, you ship a properly tested bundle, instead of trying to support all the different versions; as the end-user (unlike a developer using a lib on their own machine) should not ever have to interpret compatibility issues and should get a package that's been tested to work as a whole.

If a distro ships python 3.6 and the app wants to use 3.7, then the end result must include python 3.7 as well, either by distro being capable of having both versions at the same time or the app needs to ignore the distro-python and ship its own version in the package.

Re: Python: Please stop screwing over Linux distros

#373

Post by PeterisP »

dolmen wrote: post #55
tuldia wrote: post #18

Am I the only one that is not having issues with python and distributions in general? I get all my dependencies from Debian and they all work, when I need something that is not yet packaged, I use pip. What are people doing to get all this issues? I don't understand...

Looks like the "Works on my machine" syndrome. What about working on a project with a team? What about deploying the code on another machine?

When deploying the developed application on some server, all the exact dependencies get installed there. The main reason for the existence of the server and its configuration is to run the application, so the server adapts to the needs of the application and gets the dependency versions preferred by the app, instead of the application trying to adapt to the server and trying to make do with the libraries already existing there.

Re: Python: Please stop screwing over Linux distros

#374

Post by strbean »

1337shadow wrote: post #102
pimterry wrote: post #64

Earlier quoted context omitted.

> I always avoid pinning dependencies This is an very quick route to a maintenance nightmare imo. If you have totally unpinned dependencies, and you come back to a project after a year untouched, or 5 years, and it no longer works - which dependency update broke it? I don't agree that using an outdated package is necessarily a problem at all. Some versions are done! You don't need the latest version of every possible…

> If you have totally unpinned dependencies, and you come back to a project after a year untouched, or 5 years, and it no longer works - which dependency update broke it? In this case, it doesn't matter to me which dependency update broke it, what matters to me is to have the tests passing again with all dependencies. > I don't agree that using an outdated package is necessarily a problem at all. Some versions are do…

> In this case, it doesn't matter to me which dependency update broke it, what matters to me is to have the tests passing again with all dependencies.

Knowing which change (or changes) broke it can make resolving the issue much faster.

> If a version is done then why is there a new release? Using old versions is tech debt that will one day blow up and cost much more to correct than if it had been corrected over the time

Because was added, that is totally irrelevant to your use case.

> not to mention the security risk.

As parent mentioned, there is automated tooling for this. Your tooling yells that you are using a version of a package with a vulnerability, so you update.

> Another dependency might decide to use that dependency in a newer version, in which case aren't we all better off using the latest versions of everything?

If you don't update A, it doesn't matter if a newer version of A wants a newer version of B.

> The cost is some effort, the benefit is more features, security, performance, less bugs, basically a better program.

This assumes bugs and vulnerabilities decrease monotonically over time. This isn't true.

> Then why upgrade at all when you have a dependency with a security issue? After all, in your upgrade you might be adding even more unknown security issues that might be even more dangerous.

Because a known (to the world) security flaw is orders of magnitude more dangerous than an unknown (to the world) one, all else being equal. If there is a CVE for it, there are likely large-scale attempts at exploiting it anywhere it can be found.

> If I can test well that a newer version of a dependency works for me, why not upgrade it? There might be performance, security or other bugfixes, and I'm allowing other maintainers of other dependencies to also use that newer version.

Large amounts of labor. Furthermore, "well-tested" may include "battle tested". Some bugs make it through to deployment, and get caught and fixed. Updating dependencies without a good reason means more potential bugs slipping through, which means more bugs being discovered in deployment and a worse experience for the end user.

Re: Python: Please stop screwing over Linux distros

#375

Post by AtlasBarfed »

rcarmo wrote: post #171

I have a lot of (possibly controversial) issues about this post, for various reasons: - Debian/Ubuntu Python packaging is actually pretty comprehensive, and Ubuntu LTS ships with reasonably updated (although admittedly not latest revision) third-party packages that are usable out of the box, and that I could just list in Ansible to have usable environments spun up. - Python packaging is hardly a mess. I have been usi…

This is my experience with using any python thing on OSX:

- brew install: fails probably

- pip or pip3 or whatever: fails probably, if it succeeds, breaks something else

- look for program-specific install programs/instructions (like the aws cli for example): maybe works, probably bombs

Python's reputation (and sales pitch) among developers is a language for people that don't want to program or learn software development. It's basically the new BASIC.

Their packaging and installers only reinforce this reputation.

Not that it is sweet roses in javaland or many other language ecosystems. Dependency graphs are complicated, because they are graphs when people want them to be simple trees.

As for virtualenvs, why does a user of software that happens to be python need to know that?

Re: Python: Please stop screwing over Linux distros

#376

Post by goodpoint »

PeterisP wrote: post #371
goodpoint wrote: post #307

Earlier quoted context omitted.

Despite the downvotes, the argument stands: linux distributions are having a hard time handle the amount of tiny libraries and the conflicts in versioning and many maintainers voiced their concerns in the past years. The HN bubble can be amazing sometimes.

The point echoed in this discussion multiple times is that distributions should not handle the tiny python libraries and attempt to solve the dependency version issues, but treat an application with all its dependencies included as a single package. If a dependency needs to be bumped a version for e.g. security purposes, then the app obviously wasn't tested with the new version (which didn't exist at the time) and ne…

> If a dependency needs to be bumped a version for e.g. security purposes, then the app obviously wasn't tested with the new version (which didn't exist at the time) and needs to be retested

The burden of updating multiple copies of the same library across many packages grows exponentially and is simply untenable for distributions.

If you can find an army of volunteers to do that, distributions would love their contributions.

This hasn't happened in the last 20 years. I'd love to be proven wrong.

Re: Python: Please stop screwing over Linux distros

#377

Post by PeterisP »

goodpoint wrote: post #376
PeterisP wrote: post #371

Earlier quoted context omitted.

The point echoed in this discussion multiple times is that distributions should not handle the tiny python libraries and attempt to solve the dependency version issues, but treat an application with all its dependencies included as a single package. If a dependency needs to be bumped a version for e.g. security purposes, then the app obviously wasn't tested with the new version (which didn't exist at the time) and ne…

> If a dependency needs to be bumped a version for e.g. security purposes, then the app obviously wasn't tested with the new version (which didn't exist at the time) and needs to be retested The burden of updating multiple copies of the same library across many packages grows exponentially and is simply untenable for distributions. If you can find an army of volunteers to do that, distributions would love their contr…

Since simply updating the dependency can easily break the resulting package, this need to re-test is not something that can be avoided by making some other choice of packaging e.g. the current one - it's not adding a new burden, it's acknowledging that it already exists (indeed, IMHO much of what the original article complains about). If there are no resources to carry that burden, then the only option seems to be to wait for an updated release from the upstream, whenever that arrives.

Re: Python: Please stop screwing over Linux distros

#378

Post by rcarmo »

AtlasBarfed wrote: post #375
rcarmo wrote: post #171

I have a lot of (possibly controversial) issues about this post, for various reasons: - Debian/Ubuntu Python packaging is actually pretty comprehensive, and Ubuntu LTS ships with reasonably updated (although admittedly not latest revision) third-party packages that are usable out of the box, and that I could just list in Ansible to have usable environments spun up. - Python packaging is hardly a mess. I have been usi…

This is my experience with using any python thing on OSX: - brew install: fails probably - pip or pip3 or whatever: fails probably, if it succeeds, breaks something else - look for program-specific install programs/instructions (like the aws cli for example): maybe works, probably bombs Python's reputation (and sales pitch) among developers is a language for people that don't want to program or learn software develop…

I have the opposite experience. But you are conflating brew with pip, and they are largely independent even if you use pyenv from brew (which you should).

Again, I don’t see a lot of logic or understanding of the toolchain in that comment.

Re: Python: Please stop screwing over Linux distros

#379

Post by goodpoint »

PeterisP wrote: post #377
goodpoint wrote: post #376

Earlier quoted context omitted.

> If a dependency needs to be bumped a version for e.g. security purposes, then the app obviously wasn't tested with the new version (which didn't exist at the time) and needs to be retested The burden of updating multiple copies of the same library across many packages grows exponentially and is simply untenable for distributions. If you can find an army of volunteers to do that, distributions would love their contr…

Since simply updating the dependency can easily break the resulting package, this need to re-test is not something that can be avoided by making some other choice of packaging e.g. the current one - it's not adding a new burden, it's acknowledging that it already exists (indeed, IMHO much of what the original article complains about). If there are no resources to carry that burden, then the only option seems to be to…

I wrote "The burden of updating". Testing still needs to be done but there's a lot of automation to minimize the workload.

> If there are no resources to carry that burden, then the only option seems to be to wait for an updated release from the upstream, whenever that arrives.

No, most upstreams do not backport security fixes. And switching to a newer release is not an option if you want to provide stability to users.

Re: Python: Please stop screwing over Linux distros

#380

Post by mtrovo »

The more I see package systems that don't work the more I think Java from all the criticism got it right. Between always enforcing backward compatibility to being explicit about where to look for dependencies everything just works.

Of course you still have problems on how to declare those dependencies and resolve upgrades but a code you compiled 10 years ago will probably still work today. Compare that to Python or even worse nodejs and it's bonkers the amount of context you have to be aware of just to make your code that worked fine 2 years ago build again on the current tool chain.

edit: typos

Post reply on HN