Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PyPI Blog: Releases now reject new files after 14 days

blog.pypi.org

Post reply on HN
Print view View original

21–30 of 61 posts

Re: PyPI Blog: Releases now reject new files after 14 days

#21

Post by woodruffw »

stackskipton wrote: post #19
simonw wrote: post #17

Earlier quoted context omitted.

Published files within a release are immutable. The time limit is needed because a release can contain different binary wheels for different architectures. Consider the simplest case: your releases go out via GitHub Actions and separate wheels are built on the Windows, Linux, and macOS runners. Those won't all end at exactly the same time, so you need a release window during which they can finish and upload their gen…

Seems like better setup would be you stage a release and upload but once it’s promoted, its immutable.

Yes, that would be a better setup and it’s exactly what PyPI is moving to[1]. This closes a gap in the legacy upload API, which evolved over time instead of being designed to avoid these kinds of problems.

[1]: https://peps.python.org/pep-0694/

Re: PyPI Blog: Releases now reject new files after 14 days

#22

Post by woodruffw »

winstonwinston wrote: post #12
PhilipRoman wrote: post #2

There seems to be a severe lack of hash pinning in "modern" software ecosystems. We figured out how to do it 20+ years ago with Git bringing hash addressed storage to the masses. Coming from a different background it was very surprising for me to see things like docker images, packages and github actions being updated at the whim of upstream registry. I much prefer the philosophy where builds are fully offline and pr…

Hash pinning (already) works, and this change is all about when you as a PyPI user do not use hash pinning for installing releases, when you pin just release version for example. The release consists of one sdist and zero or more wheels. Until now you were able to upload additional wheels at later time.

Very pedantic of me, but I figure it’s interesting to note: technically a release on PyPI can have zero files or even one or more wheels but no sdist. The former is a degenerate case that users don’t normally see, and the latter happens if the user chooses to only upload wheels (or their sdist upload fails for whatever reason).

(This doesn’t change your observations at all! Just as a demonstration of how Python packaging’s data model can be unintuitive.)

Re: PyPI Blog: Releases now reject new files after 14 days

#23

Post by nneonneo »

The remaining risk now is that a patient, malicious actor could put out a new, clean source-only release, wait for ~7 days for people to decide it's safe and update to that version (and pass typical update delay controls), and then attach a bunch of malicious binary wheels. 14 days still seems to be too long.

Of course, this is already miles better than the current state of affairs where an old but popular package could become an infection vector at any time.

Re: PyPI Blog: Releases now reject new files after 14 days

#26

Post by kapilvt »

ascendantlogic wrote: post #25

Kinda curious why releases just aren't fully immutable? Sane semver would dictate any update should at least be a new patch release.

The files in a release are immutable, but a release on pypi consists of multiple files for binaries that is a cross of architecture, os, and python version. Per other comments the upload api is stateless. The consideration is an attacker adding new files to an old release

Re: PyPI Blog: Releases now reject new files after 14 days

#28

Post by akerl_ »

firesteelrain wrote: post #10

This seems like common sense configuration management 101. If I download v1.2 and it’s been published then it should be considered released and not modifiable. With exceptions for ‘dev’ releases of course. I have never published anything on PyPI but I would expect there is a publish button and finalize (?) optional button that if not checked after 14 days makes it final ?

Are there any package managers that have that kind of publish/finalize flow?

Every one I’m aware of works either as a one-shot (you have to submit everything in one push) or lets you keep adding new assets forever (other, obviously, than PyPI with the addition of this 14 day wall).

Re: PyPI Blog: Releases now reject new files after 14 days

#29

Post by akerl_ »

woodruffw wrote: post #9
lexicality wrote: post #8

14 days still seems way too long to me. As a user I thought releases on pypi were immutable!

Files are immutable on PyPI, releases are not (because releases are comprised of a set of files, and files are uploaded one by one). This is unintuitive, but the TL;DR is that files will never change on PyPI, but (previously) a user could upload a new file to a release years after their last upload to that release. This has some legitimate use cases (like allowing people to support new Python versions without bumping…

Worth noting, because I think it’s confusing some folks on this comments page, that “file” here is “a wheel for a platform” or “a complete sdist”, not “each individual .py file in a release”

You can’t go back later and add “evil.py” to a bunch of existing release files, but you could previously go find a bunch of releases that didn’t have arm64 files, publish malicious ones, and use that to catch people using those versions on arm64 systems

Re: PyPI Blog: Releases now reject new files after 14 days

#30

Post by jolmg »

kapilvt wrote: post #26
ascendantlogic wrote: post #25

Kinda curious why releases just aren't fully immutable? Sane semver would dictate any update should at least be a new patch release.

The files in a release are immutable, but a release on pypi consists of multiple files for binaries that is a cross of architecture, os, and python version. Per other comments the upload api is stateless. The consideration is an attacker adding new files to an old release

GP wasn't asking about the files being immutable, whatever a mutable file would mean. They were asking about the releases. I'm curious too.

The way it's done with rubygems, if you messed something up with the gem (pushed secrets, etc.), you "yank" (remove) the release and push a new one (different version). You can't work with the files in a release once it's been pushed.

Post reply on HN