How to improve Python packaging • Hacker News Board

Re: How to improve Python packaging

#201

Post by poulpy123 » Wed, Jan 18, 2023, 7:05 AM UTC

u801e wrote: post #186

poulpy123 wrote: post #182

Earlier quoted context omitted.

because where you have to build 1 or 2 packages for windows and macOS (97% of the computers used by end users), you have to build tens of packages for the main linux distribs (and not even all)

> you have to build tens of packages for the main linux distribs (and not even all) This is something that's typically handled by the package maintainers for each linux distro, rather than the ones who developed the application. Some developers maintain their own public repositories for packages they built and instruct end users to add their repositories to their package manager config, but they typically will also i…

I just checked and there are 1.3 millions packages in npm, 600k in pypi and 100k in nuget. While I'm sure that most of them may be either obsolete or useless, that's still an order of magnitude bigger than the packages proposed by a Linux distributions (60k for Ubuntu)

And virtualbox is actually a good exemple of what I'm saying: despite being one of the major software in one of the field where Linux is strong if not dominant.

- they feel obliged to distribute themselves their Linux packages - they have to distribute 1 package for windows, 2 for macOS but 12 for Linux.

Re: How to improve Python packaging

#202

Post by matsemann » Thu, Jan 19, 2023, 11:53 AM UTC

goodpoint wrote: post #167

matsemann wrote: post #156

Earlier quoted context omitted.

Well, it's probably more secure than doing `pip install` and letting every dependency run whatever to install itself on your host...

Actually not at all assuming you are using a VM for a production workload. Additionally, even if you only rely on a container - still no: other container engines have a lower attack surface.

I'm talking about local dev here.

Re: How to improve Python packaging

#203

Post by Yoric » Sat, Jan 21, 2023, 1:42 PM UTC

pjmlp wrote: post #151

marsten wrote: post #99

Earlier quoted context omitted.

> there will likely be a higher than average representation from people who would sit in the 10% And also an under-representation of "average users". Python the language has always benefitted from its relative simplicity, which I attribute to people like GVR saying "no" to specialized features that accrue on languages like barnacles on a ship (looking at you C++). With newer languages we see core design teams being m…

I know Python since version 1.6, its simplicity is deceptive, the language is as rich as C++, beginners only think otherwise because the language looks simple on the surface. When one starts exploring the standard library, everything that is exposed in meta-programming and runtime internals, or the usual language changes even across minor versions, the picture changes dramatically. Once upon a time I always devoured…

A few years ago, I worked on a (company-strategic) experimental project that used Python. Apparently, it only worked on some specific versions of Python, due to changes in the internal order of sets, a fact that was of course not documented or checked in the tool.

Not the reason for which the project failed, but the weeks lost trying to replicate results between teams certainly didn't help.

Python is very readable, but I agree that the stdlib is anything but simple.

Re: How to improve Python packaging

#204

Post by u801e » Sun, Jan 22, 2023, 5:44 AM UTC

poulpy123 wrote: post #201

u801e wrote: post #186

Earlier quoted context omitted.

> you have to build tens of packages for the main linux distribs (and not even all) This is something that's typically handled by the package maintainers for each linux distro, rather than the ones who developed the application. Some developers maintain their own public repositories for packages they built and instruct end users to add their repositories to their package manager config, but they typically will also i…

I just checked and there are 1.3 millions packages in npm, 600k in pypi and 100k in nuget. While I'm sure that most of them may be either obsolete or useless, that's still an order of magnitude bigger than the packages proposed by a Linux distributions (60k for Ubuntu) And virtualbox is actually a good exemple of what I'm saying: despite being one of the major software in one of the field where Linux is strong if not…

I'm not sure what goes into the decisions package maintainers make in terms of which packages to include in the OS repository, but, in my experience with python applications, most dependencies we needed could be found in the OS repository or some other supported package repositories (for RPM, repositories like epel or rpmfusion). The few we weren't able to find weren't that difficult to package and add to our internal package repository.

But this also brings up the issue of vetting dependencies. If you're pulling in a dependency that pulls in 10s of other dependencies (direct and indirect), it gets difficult to vet them. PyPi and npm have already had issues with malicious packages being uploaded. On the other hand, I haven't really found the large number of dependencies being an issue for python packages available in the OS package repositories, and I'm not aware of any incidents with those repositories unlike PyPi and npm.