Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PyPI Blog: Releases now reject new files after 14 days

blog.pypi.org

Post reply on HN
Print view View original

51–60 of 61 posts

Re: PyPI Blog: Releases now reject new files after 14 days

#51

Post by firesteelrain »

akerl_ wrote: post #41
firesteelrain wrote: post #31

Earlier quoted context omitted.

Doesn’t that sound bizarre? I have never heard of such a thing. The builds should be immutable

No? Again, can you think of any packages managers that have a finalize step like you’re describing? All the package managers I’m aware of do one of two things: 1. You push once with everything baked in. 2. You push as many things as you want forever. Python releases can sometimes have many different package files (for example, because each platform can have its own wheel), which makes the first option pretty challeng…

My point is adding stuff to an existing version doesn’t make it immutable like it should such as Maven-style

Re: PyPI Blog: Releases now reject new files after 14 days

#52

Post by [deleted] »

simonw wrote: post #17
skinfaxi wrote: post #14

14 days is still too long if you ask me. Releases should be immutable.

Published files within a release are immutable. The time limit is needed because a release can contain different binary wheels for different architectures. Consider the simplest case: your releases go out via GitHub Actions and separate wheels are built on the Windows, Linux, and macOS runners. Those won't all end at exactly the same time, so you need a release window during which they can finish and upload their gen…

[deleted]

Re: PyPI Blog: Releases now reject new files after 14 days

#53

Post by [deleted] »

simonw wrote: post #17
skinfaxi wrote: post #14

14 days is still too long if you ask me. Releases should be immutable.

Published files within a release are immutable. The time limit is needed because a release can contain different binary wheels for different architectures. Consider the simplest case: your releases go out via GitHub Actions and separate wheels are built on the Windows, Linux, and macOS runners. Those won't all end at exactly the same time, so you need a release window during which they can finish and upload their gen…

[deleted]

Re: PyPI Blog: Releases now reject new files after 14 days

#54

Post by woodruffw »

PunchyHamster wrote: post #34
woodruffw wrote: post #9

Earlier quoted context omitted.

Files are immutable on PyPI, releases are not (because releases are comprised of a set of files, and files are uploaded one by one). This is unintuitive, but the TL;DR is that files will never change on PyPI, but (previously) a user could upload a new file to a release years after their last upload to that release. This has some legitimate use cases (like allowing people to support new Python versions without bumping…

> (like allowing people to support new Python versions without bumping a package’s version) ...why Python is just breaking compatibility so bad with new version it needs that ?

It doesn’t. It’s really only relevant if you use version-specific APIs/ABIs in wheels containing machine code. Pure Python wheels don’t generally need this.

Re: PyPI Blog: Releases now reject new files after 14 days

#55

Post by woodruffw »

crabbone wrote: post #43
woodruffw wrote: post #6

Earlier quoted context omitted.

I’m not sure what this has to do with TFA: Python does have hash-pinning. TFA is not about modifying existing files on the index (PyPI doesn’t allow that), but about adding new files to a pre-existing release. But that doesn’t change the hash of older distributions on that release.

1. There are multiple levels at which things are hashed, signed and checked when it comes to Python packages. Eg. each file in the Wheel beside the RECORD file is hashed using SHA256. This hash is never checked :( You can also sign your packages (RECORD.jws anyone? Is that still supported?), but nobody checks that either. 2. There are hashes in the HTML served by PyPI. These are updated at the whim of both the index…

There are a lot of misconceptions here:

1. “Wheel” signatures (e.g. RECORD.jws) were specified but never actually implemented in any major packaging installer, to the best of my knowledge. In that sense it’s not really meaningful to ask whether they’re “still” supported, since nothing ever did support them unfortunately. The reasons for that are multifaceted, but a single easily identifiable reason is that they don’t solve the key/identity distribution problem.

2. The primary benefit of index-supplied hashing is for lockfiles. Specifically, they make lockfiles like uv.lock and pylock.toml useful/meaningful.

3. Is the source distribution problem, and is sort of fundamental to any packaging system that has any amount of dynamism in it. There’s a good argument to be made for reducing unnecessary dynamism (e.g. via wheels), but a nontrivial percent of Python users actually want this behavior.

Re: files: I think you’re getting confused about distribution files (sdists and wheels) versus the individual contents therein. You can’t modify distribution files on PyPI, you can only add new, unique ones. And this is now restricted to 14 days for each release.

Re: PyPI Blog: Releases now reject new files after 14 days

#56

Post by nobodyandproud »

joombaga wrote: post #16
nobodyandproud wrote: post #13

Earlier quoted context omitted.

I feel you’re quibbling over semantics here. In concept why can’t the full set of files in a release be a single, one-way hash value, with both adding or releasing changing the hash value?

The semantics are important. A release is composed of multiple distributions (sets of files). Each distribution does have a one-way hash value. This is what you lock to. Adding files does change the value of that hash. Files cannot be removed or changed. You lock to the distribution instead of the release so your build doesn't have to download distributions for platforms you're not using.

It’s not. Steel manning the OP (which at face value seems to be the most reasonable interpretation) they’re wondering why an entire release isn’t hash pinned.

The response is that there is hash pinning but only at the constituent levels. Which clearly isn’t what the OP meant.

Re: PyPI Blog: Releases now reject new files after 14 days

#57

Post by nobodyandproud »

woodruffw wrote: post #20
nobodyandproud wrote: post #13

Earlier quoted context omitted.

I feel you’re quibbling over semantics here. In concept why can’t the full set of files in a release be a single, one-way hash value, with both adding or releasing changing the hash value?

Because uploads to PyPI are not atomic. They’re now capped within a 14 day window, but it would be extremely confusing to users to have the “release” hash of their dependencies change repeatedly. (Also: we’d need to determine what it even means to hash a set of files. Do we order by canonicalized filename, by upload time, etc. Each of these has surprising implications!) Edit: to be clear, it’s not ideal or fully inte…

> Because uploads to PyPI are not atomic. They’re now capped within a 14 day window, but it would be extremely confusing to users to have the “release” hash of their dependencies change repeatedly.

Aren’t releases versioned, using semantic versioning?

If not, why shouldn’t a change in dependencies not trigger a new release version?

If it is versioned and if there’s a corresponding hash to the release, why wouldn’t I also expect that to change when the version and dependencies change?

I can understand that there may be historical reasons that this scheme will break how PyPI releases are built and distributed, but I also hope you understand that it violates the principle of least surprise.

Re: PyPI Blog: Releases now reject new files after 14 days

#58

Post by woodruffw »

nobodyandproud wrote: post #57
woodruffw wrote: post #20

Earlier quoted context omitted.

Because uploads to PyPI are not atomic. They’re now capped within a 14 day window, but it would be extremely confusing to users to have the “release” hash of their dependencies change repeatedly. (Also: we’d need to determine what it even means to hash a set of files. Do we order by canonicalized filename, by upload time, etc. Each of these has surprising implications!) Edit: to be clear, it’s not ideal or fully inte…

> Because uploads to PyPI are not atomic. They’re now capped within a 14 day window, but it would be extremely confusing to users to have the “release” hash of their dependencies change repeatedly. Aren’t releases versioned, using semantic versioning? If not, why shouldn’t a change in dependencies not trigger a new release version? If it is versioned and if there’s a corresponding hash to the release, why wouldn’t I…

Python uses PEP 440 versioning, which isn’t semver. Semver is de facto common, however, and can be expressed as a subset of PEP 440.

I’m not sure what you mean by “change in dependencies”: every release has zero or more distributions (“files”), and each distribution in a release can specify its own dependencies. For example, a macOS-specific wheel might depend on something that Linux-specific wheels don’t need or vice versa.

> but I also hope you understand that it violates the principle of least surprise.

I think it’s fair to say that virtually everything about Python packaging violates POLA :-)

Re: PyPI Blog: Releases now reject new files after 14 days

#59

Post by ndndndb »

flakes wrote: post #48
yladiz wrote: post #3

I’m a bit surprised this is possible in the first place. I get that you might not be able to upload everything in one go, but it feels like you should “start” and “finish” a release in that case, and once it’s finished you can’t modify it. I guess the use case is that you might want to build a wheel for an older release for a newer version of Python?

IMO a release workflow would be better with something like this: 1) Upload all files in a staging state. Can be done asynchronously via multiple build hosts. Files in this state are referenced via their cryptographic checksums (e.g. SHA, Blake, etc) 2) Make visible with a single call by providing a manifest with checksums for all source dists/wheels contained in the release. All artifacts are made visible atomically…

Yea that would be the obvious implementation. No idea why they didn't choose that route

Re: PyPI Blog: Releases now reject new files after 14 days

#60

Post by gred »

akerl_ wrote: post #49
gred wrote: post #47

Earlier quoted context omitted.

Yeah, I think that's the difference. In Maven, the entire set of files which compose a release is immutable. You can't add to or remove from the set of files once you've published. You have to release a new version if you want to add anything.

Interesting. I’ve only used Maven-format repos once or twice and always via somebody’s preexisting CI tooling. Is there essentially a “draft” release that you can edit repeatedly before hitting “publish”? Is that draft release visible externally as a pre-release, or only to the author?

There is indeed a "draft" release (I think they call it a "deployment"), but AFAIK your choice is then to either publish it or delete it. If you review it and it looks good, you publish it and can't change it thereafter (no changes to files, no adding new files, no removing existing files). If, however, you need to change something in the deployment before publishing it, you have to delete the deployment and create + upload a new "draft" release / deployment (can be same version number, since the previous "draft" wasn't published). AFAIK these "draft" releases are visible only to the author.
Post reply on HN