Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pypi.org is running a survey on the state of Python packaging

pypi.org

Post reply on HN
Print view View original

171–180 of 193 posts

Re: Pypi.org is running a survey on the state of Python packaging

#171

Post by Const-me »

chlorion wrote: post #129
cycomanic wrote: post #99

Earlier quoted context omitted.

I know this is unpopular opinion on here, but I believe all this packaging madness is forced on us by languages because Windows (and to a lesser degree osx) have essentially no package management. Especially installing a tool chain to compile C code for python is no issue on Linux, but such a pain on Windows.

It may be unpopular but it's correct! Every language tries to re-implement the package manager, but it ends up breaking down as soon as you need to interact with anything outside of that specific language's ecosystem. The only solution for interacting with the "outside" (other languages, toolchains, etc) is a system level, language agnostic package manager of some kind. Linux distros package management is far from pe…

> it ends up breaking down as soon as you need to interact with anything outside of that specific language's ecosystem.

It works OK in nuget on Windows, due to the different approach taken by the OS maintainer.

A DLL compiled 25 years ago for Windows NT stills work on a modern Windows, as long as the process is 32-bit. A DLL compiled 15 years ago for 64-bit Vista will still work on a modern Windows without issues at all.

People who need native code in their nuget packages are simply shipping native DLLs, often with static link to C runtime. Probably the most popular example of such package is SQLite.

> I very highly recommend people to learn how to write and create Linux packages if they need to distribute software.

I agree. When working on embedded Linux software where I control the environment and don’t care about compatibility with different Linuxes or different CPU architectures, I often package my software into *.deb packages.

Re: Pypi.org is running a survey on the state of Python packaging

#172

Post by siproprio »

klkqw wrote: post #112
siproprio wrote: post #77

State of the art python packaging must include support for common use cases such as conda+machine learning. It’s incredible how even Julia’s Pkg.jl supports better python packaging in combination with conda than the official python packaging tools. This is very clearly a question of the culture of the core python developers (such as brett cannon) who seem to think the machine learning people with their compilers and…

The Twitter hero faction of the Python core "developers" (many of whom have not done much actual work for a decade) has always been pro-web and anti-C-extensions. They have been peddling the dream of speeding up Python by neglecting the scientific ecosystem for a long time without having either the expertise or a clear plan how to achieve that. All that while the primarily useful part of Python is its C extension cap…

The pypy folks and a bunch of other alternative python implementations have pooled together and are working on a soft fork of the c extension api to enable existing established projects that make heavy use of the c api to better work with the superior faster python implementations.

I believe this is the way forward. Existing packages are essentially tied to cpython because that’s what the python.org website offers for download, and people will just use that by default.

Once all major packages support the new c api, it will be easier to just use something other than cpython by default.

Re: Pypi.org is running a survey on the state of Python packaging

#173

Post by BerislavLopac »

duckmysick wrote: post #168
BerislavLopac wrote: post #161

Earlier quoted context omitted.

I'm not sure what you mean - unlike npm, pip installs only one copy of a dependency in each environment.

So if I have environments A, B, and C and each has a dependency ML-package-1.2 that's 100 MB, it means there's one copy of it in each environment? Meaning 3x100 MB?

Ah, so across multiple environments? You do have a point, but that has nothing to do with pip - it's not an environment manager.

I can see that as an option in tools like venv or poetry, possibly setting a "common" location to share across environments.

Re: Pypi.org is running a survey on the state of Python packaging

#174

Post by jbylund »

LukeShu wrote: post #111
orf wrote: post #49

Earlier quoted context omitted.

Yeah, that’s sort of what I meant by my suggestion. Requirements that can only be resolved by downloading and executing code is a huge burden on tooling

If the package is available as a wheel, you don't need to execute code to see what the requirements are; you just need to parse the "METADATA" file. However, the only way to get the METADATA for a wheel (using PyPA standard APIs, anyway) is to download the whole wheel. For comparison, pacman (the Arch Linux package manager) packages have fairly similar ".PKGINFO" file in them; but in order to support resolving depend…

Pip can use range requests to fetch just a part of the wheel, and lift the metadata out of that. So it can sometimes avoid downloading the entire wheel just to get the deps. Some package servers don't support this though.

Also, there's a difference between a pep being adopted and that pep being implemented (usually a bunch of elbow grease). That said there are a couple exciting steps towards 658 being implemented: https://github.com/pypa/pip/pull/11111 (just approved yesterday, not yet merged) https://github.com/pypi/warehouse/issues/8254 (been open for forever, but there has been incremental progress made. Warehouse seems to not attract the same amount of contribution as pip)

Re: Pypi.org is running a survey on the state of Python packaging

#175

Post by jbylund »

mdmglr wrote: post #17

My wishlist: We need a way to configure an ordered list of indexes pip searches for packages. —extra-index-url or using a proxy index is not the solution. Also namespaces and not based on a domain. So for example: pip install apache:parquet Also some logic either in the pip client or index server to minimize typosquatting Also pip should adopt a lock file similar to npm/yarn. Instead of requirements.txt And also “pip…

Typosquatting is a thing that has been looked at and is being looked at:

https://github.com/pypi/warehouse/pull/5001 - had to be reverted because it was too noisy

https://github.com/pypi/warehouse/issues/9527

Re: Pypi.org is running a survey on the state of Python packaging

#176

Post by dalke »

Spiritus wrote: post #170
dalke wrote: post #165

Earlier quoted context omitted.

How do I specify that I need gfortran installed?

You can't. But is that possible with any programming language specific package manager? How would that even work given that every flavour of OS/distro have their own way of providing gfortran?

You can't. But my g'parent comment in this thread was because my Python module needs the OpenMP library, or compile-time detection that it wasn't there, to skip OpenMP support. The latter is done by an environment variable which my setup.py understands.

Then orf dreamed of a day where you could "describe what system libraries I may need without having to execute setup.py and find out, and express compile time flags."

The link you pointed doesn't appear to handle what we were talking about. By specifying "gfortran", I hoped to highlight that difference.

riff, building on nix, seems an intriguing solution for this.

Re: Pypi.org is running a survey on the state of Python packaging

#177

Post by pletnes »

BerislavLopac wrote: post #160
pletnes wrote: post #148

Earlier quoted context omitted.

It’s only the last 5 years, give or take, that you can install C-compiled packages with pip. When I started out, I had to apt/port/pacman install packages, then run pip install (or python setup.py even) to install dependencies. It’s hard for newer python converts to even comprehend the pain it used to be. Conda came with the compiled package concept, and pip came after. Dealing with dependencies these days is a breez…

> It’s only the last 5 years, give or take, that you can install C-compiled packages with pip Give, quite a bit. The egg format was introduced in 2004 [0], and even the newer wheel, that replaced it, is turning 10 later this month. [0] https://packaging.python.org/en/latest/discussions/wheel-vs-...

I started using numpy around 2007 and didn’t see reliable binary installs with good numerical performance until I discovered anaconda, much later (2016?). Maybe I hung out with the wrong people, or libraries. Some libraries did not have compiled wheels a few years ago, and M1 macs still run into the very occasional issue.

Re: Pypi.org is running a survey on the state of Python packaging

#178

Post by pletnes »

BerislavLopac wrote: post #160
pletnes wrote: post #148

Earlier quoted context omitted.

It’s only the last 5 years, give or take, that you can install C-compiled packages with pip. When I started out, I had to apt/port/pacman install packages, then run pip install (or python setup.py even) to install dependencies. It’s hard for newer python converts to even comprehend the pain it used to be. Conda came with the compiled package concept, and pip came after. Dealing with dependencies these days is a breez…

> It’s only the last 5 years, give or take, that you can install C-compiled packages with pip Give, quite a bit. The egg format was introduced in 2004 [0], and even the newer wheel, that replaced it, is turning 10 later this month. [0] https://packaging.python.org/en/latest/discussions/wheel-vs-...

9 out of the top 360 packages still do not offer wheels! https://pythonwheels.com/

Re: Pypi.org is running a survey on the state of Python packaging

#179

Post by dagw »

BerislavLopac wrote: post #160
pletnes wrote: post #148

Earlier quoted context omitted.

It’s only the last 5 years, give or take, that you can install C-compiled packages with pip. When I started out, I had to apt/port/pacman install packages, then run pip install (or python setup.py even) to install dependencies. It’s hard for newer python converts to even comprehend the pain it used to be. Conda came with the compiled package concept, and pip came after. Dealing with dependencies these days is a breez…

> It’s only the last 5 years, give or take, that you can install C-compiled packages with pip Give, quite a bit. The egg format was introduced in 2004 [0], and even the newer wheel, that replaced it, is turning 10 later this month. [0] https://packaging.python.org/en/latest/discussions/wheel-vs-...

Just because the egg/wheel format existed, didn't mean that they where actually used and actually worked. Being able to reliable pip install packages like numpy and gdal only started 3-4 years ago. There is a reason that Christoph Gohlke's python package site exists and was popular.

Re: Pypi.org is running a survey on the state of Python packaging

#180

Post by dec0dedab0de »

EdwardDiego wrote: post #156
dec0dedab0de wrote: post #127

Earlier quoted context omitted.

Pipenv is pretty nice, I still use it. Kenneth Reitz really has a knack for interfaces and making things easier for developers to use. The fiasco was that he was not the best at maintaining projects, and used his popularity from tablib and requests to get pipenv recommended by the PyPA well before it was ready for general use. Then around the same time there was a few scandals I don't remember the details of, somethi…

I was trying to to use Pipenv around the time of the drama. Ended up switching to Poetry because it had a superior dependency resolution algorithm.

Ugh I remember the dependency problems. I think we worked around it by explicitly installing dependencies first. We did almost switch to poetry because of that, but it came down to "everyone is busy, lets just see how often it is a problem". Luckily we didn't run into it too many times, and they have since fixed it.
Post reply on HN