Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to improve Python packaging

chriswarrick.com

Post reply on HN
Print view View original

181–190 of 204 posts

Re: How to improve Python packaging

#181

Post by groestl »

u801e wrote: post #177
LtWorf wrote: post #56

Earlier quoted context omitted.

.deb packages…

On that note, why are we trying to use language specific packaging tools to build packages rather than just building OS specific packages where dependencies are handled by apt for deb packages or dnf for rpm packages? These are language agnostic and can get the job done.

Because I have dozens of colleagues, each with their own os/distribution, developing an application that needs to lock down dot releases of dependencies to the versions running in production and update them at the same time. There is no way to do that using distribution specific tools without going insane.

Re: How to improve Python packaging

#182

Post by poulpy123 »

u801e wrote: post #177
LtWorf wrote: post #56

Earlier quoted context omitted.

.deb packages…

On that note, why are we trying to use language specific packaging tools to build packages rather than just building OS specific packages where dependencies are handled by apt for deb packages or dnf for rpm packages? These are language agnostic and can get the job done.

because where you have to build 1 or 2 packages for windows and macOS (97% of the computers used by end users), you have to build tens of packages for the main linux distribs (and not even all)

Re: How to improve Python packaging

#183

Post by u801e »

groestl wrote: post #181
u801e wrote: post #177

Earlier quoted context omitted.

On that note, why are we trying to use language specific packaging tools to build packages rather than just building OS specific packages where dependencies are handled by apt for deb packages or dnf for rpm packages? These are language agnostic and can get the job done.

Because I have dozens of colleagues, each with their own os/distribution, developing an application that needs to lock down dot releases of dependencies to the versions running in production and update them at the same time. There is no way to do that using distribution specific tools without going insane.

>> why are we trying to use language specific packaging tools to build packages rather than just building OS specific packages where dependencies are handled by apt for deb packages or dnf for rpm packages?

>>

>> These are language agnostic and can get the job done.

> Because I have dozens of colleagues, each with their own os/distribution, developing an application that needs to lock down dot releases of dependencies to the versions running in production

I assume that the production environment isn't running dozens of os versions and distributions. For development, using a VM or container running the same os version and distro that's used in production and using that OS's package format for packaging the software and installing it in the dev environment (on the container or VM) would work. You're testing to see if the software works in the production environment, not someone's preferred os/distribution.

> and update them at the same time

I'm not as familiar with apt, but dnf has a version lock feature that would allow you to lock down the dependencies to specific versions. You could update them and test them during development and update the version lock file to pull down the updated version of the dependency when updating production.

Re: How to improve Python packaging

#184

Post by groestl »

LtWorf wrote: post #180
groestl wrote: post #98

Earlier quoted context omitted.

Do you happen to have a link to a python application that ships it's own dependencies and has state-of-the art debian packaging?

> that ships it's own dependencies You mean a link to a malformed .deb packages? It's not difficult to do but it's also the wrong way to do it.

Honestly, there is a lot more I'd like to do with my life than caring about some debian packaging practices when all I want to do is ship a company's internal application with it's exact dependency versions. But it needs to work, and, e.g. "The virtualenv relocate-able flag has been always experimental, and never really worked;"

So in practice we're back to shipping whole distributions using Docker (or, like we did, abandon Python as an option for development but keep .deb packages for deployment).

Re: How to improve Python packaging

#185

Post by groestl »

u801e wrote: post #183
groestl wrote: post #181

Earlier quoted context omitted.

Because I have dozens of colleagues, each with their own os/distribution, developing an application that needs to lock down dot releases of dependencies to the versions running in production and update them at the same time. There is no way to do that using distribution specific tools without going insane.

>> why are we trying to use language specific packaging tools to build packages rather than just building OS specific packages where dependencies are handled by apt for deb packages or dnf for rpm packages? >> >> These are language agnostic and can get the job done. > Because I have dozens of colleagues, each with their own os/distribution, developing an application that needs to lock down dot releases of dependencie…

For every dependency that's missing from the upstream distribution in the exact version we need, we'd need to package that appropriately. We have nothing to gain here, nobody pays for that.

Re: How to improve Python packaging

#186

Post by u801e »

poulpy123 wrote: post #182
u801e wrote: post #177

Earlier quoted context omitted.

On that note, why are we trying to use language specific packaging tools to build packages rather than just building OS specific packages where dependencies are handled by apt for deb packages or dnf for rpm packages? These are language agnostic and can get the job done.

because where you have to build 1 or 2 packages for windows and macOS (97% of the computers used by end users), you have to build tens of packages for the main linux distribs (and not even all)

> you have to build tens of packages for the main linux distribs (and not even all)

This is something that's typically handled by the package maintainers for each linux distro, rather than the ones who developed the application. Some developers maintain their own public repositories for packages they built and instruct end users to add their repositories to their package manager config, but they typically will also include the source archive (if it's open source) and build instructions for those running distributions they haven't built packages for.

Looking at Virtualbox[1], for example, in addition to Mac and Windows, they built packages for Redhat, Ubuntu, Debian, OpenSUSE, and Fedora, and they provided the sources along with build instructions[2] for those who are running distributions where there's no pre-built package. In fact, the last option is the most flexible one though it requires a bit more work for the end user.

[1] https://www.virtualbox.org/wiki/Linux_Downloads

[2] https://www.virtualbox.org/wiki/Downloads

Re: How to improve Python packaging

#187

Post by u801e »

groestl wrote: post #185
u801e wrote: post #183

Earlier quoted context omitted.

>> why are we trying to use language specific packaging tools to build packages rather than just building OS specific packages where dependencies are handled by apt for deb packages or dnf for rpm packages? >> >> These are language agnostic and can get the job done. > Because I have dozens of colleagues, each with their own os/distribution, developing an application that needs to lock down dot releases of dependencie…

For every dependency that's missing from the upstream distribution in the exact version we need, we'd need to package that appropriately. We have nothing to gain here, nobody pays for that.

Once you package a dependency, updating to a new version requires minor changes unless it's a major version change. What you do gain is the ability to easily upgrade and downgrade a particular dependency and verify the integrity of the installed files (something that pip, for example, doesn't provide as far as I'm aware, but the OS package manager does).

Re: How to improve Python packaging

#188

Post by groestl »

u801e wrote: post #187
groestl wrote: post #185

Earlier quoted context omitted.

For every dependency that's missing from the upstream distribution in the exact version we need, we'd need to package that appropriately. We have nothing to gain here, nobody pays for that.

Once you package a dependency, updating to a new version requires minor changes unless it's a major version change. What you do gain is the ability to easily upgrade and downgrade a particular dependency and verify the integrity of the installed files (something that pip, for example, doesn't provide as far as I'm aware, but the OS package manager does).

This is just a crazy amount of work, given the alternative is updating your whatever lock file and let the transitive dependency resolution of your language of choice do the rest. Note that I very much like proper clean .debs as an end user, and if that's my customer base I'd publish like that as well. But if I'm last in the chain and my customers are intern, I'd never ever in a million years take the route you propose.

Re: How to improve Python packaging

#189

Post by BerislavLopac »

Despite being misguided in a few points, the best bit about this article is the detailed table comparing the functionalities of different Python packaging tools.

One missing detail is the ability of PDM to support different build backends, which allows for some interesting capabilities: for example, using hatchling as the backend it is possible to utilise hatch's support for dynamic versioning, whcih does not exist in PDM proper. I haven't tried, but wouldn't be surprised if that could allow PDM to support C-extensions by using setuptools as the backend...

Re: How to improve Python packaging

#190

Post by BerislavLopac »

BerislavLopac wrote: post #189

Despite being misguided in a few points, the best bit about this article is the detailed table comparing the functionalities of different Python packaging tools. One missing detail is the ability of PDM to support different build backends, which allows for some interesting capabilities: for example, using hatchling as the backend it is possible to utilise hatch's support for dynamic versioning, whcih does not exist i…

One of the misguided points is this:

> It is also notable that PEP 20, the Zen of Python, states this:

> There should be one-- and preferably only one --obvious way to do it.

> Python packaging definitely does not follow it. There are 14 ways, and none of them is obvious or the only good one. All in all, this is an unsalvageable mess. Why can’t Python pick one tool?

So a few comments on that:

1. There is a reason PEP 20 is called "Zen of Python" and not "Zen of Python Packaging".

1a. Even if it applied to the ecosystem and not just the language itself - PEP 20 is a guide, not a set of divine laws.

2. There is one obvious way, right there in the Python documentation. [0] Yes, it lists several different tools, but "a way to do it" and "a tool to do it with" are two very different topics.

3. "Why can’t Python pick one tool?" I never understood this fixation on silver bullets and "one tool to rule them all"... As long as common principles are well defined - which has been true for Python for quite some time even before PEP 517, with things like PyPI and pip - what is the harm in having multiple competing solutions?

[0] https://packaging.python.org/en/latest/tutorials/packaging-p...

Post reply on HN