Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PyPI Blog: Releases now reject new files after 14 days

blog.pypi.org

Post reply on HN
Print view View original

41–50 of 61 posts

Re: PyPI Blog: Releases now reject new files after 14 days

#41

Post by akerl_ »

firesteelrain wrote: post #31
akerl_ wrote: post #28

Earlier quoted context omitted.

Are there any package managers that have that kind of publish/finalize flow? Every one I’m aware of works either as a one-shot (you have to submit everything in one push) or lets you keep adding new assets forever (other, obviously, than PyPI with the addition of this 14 day wall).

Doesn’t that sound bizarre? I have never heard of such a thing. The builds should be immutable

No?

Again, can you think of any packages managers that have a finalize step like you’re describing?

All the package managers I’m aware of do one of two things:

1. You push once with everything baked in.

2. You push as many things as you want forever.

Python releases can sometimes have many different package files (for example, because each platform can have its own wheel), which makes the first option pretty challenging.

Re: PyPI Blog: Releases now reject new files after 14 days

#42

Post by etbebl »

akerl_ wrote: post #41
firesteelrain wrote: post #31

Earlier quoted context omitted.

Doesn’t that sound bizarre? I have never heard of such a thing. The builds should be immutable

No? Again, can you think of any packages managers that have a finalize step like you’re describing? All the package managers I’m aware of do one of two things: 1. You push once with everything baked in. 2. You push as many things as you want forever. Python releases can sometimes have many different package files (for example, because each platform can have its own wheel), which makes the first option pretty challeng…

I'm learning here, but for option 2, besides the security risk, doesn't this create the possibility that users could get a broken/unfinished package any time they're updating to a recently "published" release? The property of releases being atomic seems very important.

Re: PyPI Blog: Releases now reject new files after 14 days

#43

Post by crabbone »

woodruffw wrote: post #6
PhilipRoman wrote: post #2

There seems to be a severe lack of hash pinning in "modern" software ecosystems. We figured out how to do it 20+ years ago with Git bringing hash addressed storage to the masses. Coming from a different background it was very surprising for me to see things like docker images, packages and github actions being updated at the whim of upstream registry. I much prefer the philosophy where builds are fully offline and pr…

I’m not sure what this has to do with TFA: Python does have hash-pinning. TFA is not about modifying existing files on the index (PyPI doesn’t allow that), but about adding new files to a pre-existing release. But that doesn’t change the hash of older distributions on that release.

1. There are multiple levels at which things are hashed, signed and checked when it comes to Python packages. Eg. each file in the Wheel beside the RECORD file is hashed using SHA256. This hash is never checked :( You can also sign your packages (RECORD.jws anyone? Is that still supported?), but nobody checks that either.

2. There are hashes in the HTML served by PyPI. These are updated at the whim of both the index and the publisher. Even though they are checked by pip during install, they are worthless.

3. There are many ways to install packages that work around (2). Custom index server doesn't have to provide hashes, and pip will happily install that. You can install from sources, from a package you've downloaded somewhere, form VCS, you can build it during install, all without even prompting the user to confirm the very scary choices.

NB. I have no idea how do you make the leap from "adding files to release" to "not modifying the release". To me, adding file to release is sure as hell modifying it. Here's a very simple malicious example:

I release package "innocent" with an empty "scripts" section. Then, in the subsequent modification to this release, I add the "scripts" section with a script named "notebook". Now, whenever my user wants to run Jupyter notebook, they will call my "notebook" program, not the one from Jupyter package.

Re: PyPI Blog: Releases now reject new files after 14 days

#44

Post by gred »

akerl_ wrote: post #28
firesteelrain wrote: post #10

This seems like common sense configuration management 101. If I download v1.2 and it’s been published then it should be considered released and not modifiable. With exceptions for ‘dev’ releases of course. I have never published anything on PyPI but I would expect there is a publish button and finalize (?) optional button that if not checked after 14 days makes it final ?

Are there any package managers that have that kind of publish/finalize flow? Every one I’m aware of works either as a one-shot (you have to submit everything in one push) or lets you keep adding new assets forever (other, obviously, than PyPI with the addition of this 14 day wall).

In the Java world, Maven has a "publish" step. Published artifacts (groups of files) are immutable, so publish == finalize.

Re: PyPI Blog: Releases now reject new files after 14 days

#45

Post by akerl_ »

gred wrote: post #44
akerl_ wrote: post #28

Earlier quoted context omitted.

Are there any package managers that have that kind of publish/finalize flow? Every one I’m aware of works either as a one-shot (you have to submit everything in one push) or lets you keep adding new assets forever (other, obviously, than PyPI with the addition of this 14 day wall).

In the Java world, Maven has a "publish" step. Published artifacts (groups of files) are immutable, so publish == finalize.

That’s exactly how individual artifacts are (and were) on PyPI. This change isn’t to artifact immutability, it’s to releases (collections of artifacts)

Re: PyPI Blog: Releases now reject new files after 14 days

#46

Post by akerl_ »

etbebl wrote: post #42
akerl_ wrote: post #41

Earlier quoted context omitted.

No? Again, can you think of any packages managers that have a finalize step like you’re describing? All the package managers I’m aware of do one of two things: 1. You push once with everything baked in. 2. You push as many things as you want forever. Python releases can sometimes have many different package files (for example, because each platform can have its own wheel), which makes the first option pretty challeng…

I'm learning here, but for option 2, besides the security risk, doesn't this create the possibility that users could get a broken/unfinished package any time they're updating to a recently "published" release? The property of releases being atomic seems very important.

Generally each artifact in the release is atomic, as is (and was) the case here.

You’ll never get served a partially uploaded wheel for amd64 Linux, but somebody could come back a year later and add a 2nd wheel to that release for a different architecture.

Re: PyPI Blog: Releases now reject new files after 14 days

#47

Post by gred »

akerl_ wrote: post #45
gred wrote: post #44

Earlier quoted context omitted.

In the Java world, Maven has a "publish" step. Published artifacts (groups of files) are immutable, so publish == finalize.

That’s exactly how individual artifacts are (and were) on PyPI. This change isn’t to artifact immutability, it’s to releases (collections of artifacts)

Yeah, I think that's the difference. In Maven, the entire set of files which compose a release is immutable. You can't add to or remove from the set of files once you've published. You have to release a new version if you want to add anything.

Re: PyPI Blog: Releases now reject new files after 14 days

#48

Post by flakes »

yladiz wrote: post #3

I’m a bit surprised this is possible in the first place. I get that you might not be able to upload everything in one go, but it feels like you should “start” and “finish” a release in that case, and once it’s finished you can’t modify it. I guess the use case is that you might want to build a wheel for an older release for a newer version of Python?

IMO a release workflow would be better with something like this:

1) Upload all files in a staging state. Can be done asynchronously via multiple build hosts. Files in this state are referenced via their cryptographic checksums (e.g. SHA, Blake, etc)

2) Make visible with a single call by providing a manifest with checksums for all source dists/wheels contained in the release. All artifacts are made visible atomically and the release is immutable.

That allows authors to prepare uploads over however many days they need to coordinate hardware, but doesn’t allow for users to discover a release in a partial state.

Re: PyPI Blog: Releases now reject new files after 14 days

#49

Post by akerl_ »

gred wrote: post #47
akerl_ wrote: post #45

Earlier quoted context omitted.

That’s exactly how individual artifacts are (and were) on PyPI. This change isn’t to artifact immutability, it’s to releases (collections of artifacts)

Yeah, I think that's the difference. In Maven, the entire set of files which compose a release is immutable. You can't add to or remove from the set of files once you've published. You have to release a new version if you want to add anything.

Interesting. I’ve only used Maven-format repos once or twice and always via somebody’s preexisting CI tooling. Is there essentially a “draft” release that you can edit repeatedly before hitting “publish”? Is that draft release visible externally as a pre-release, or only to the author?

Re: PyPI Blog: Releases now reject new files after 14 days

#50

Post by simonw »

crabbone wrote: post #40
stackskipton wrote: post #19

Earlier quoted context omitted.

Seems like better setup would be you stage a release and upload but once it’s promoted, its immutable.

There's no reason a public index should allow any "staging" time for releases. It's the whole point of the release: once it's done, it's sealed. Oh, you messed up? -- Either make a new release, or release patches for the old one. Do not use production release server as your testing ground. It's not meant to do that, it makes it unnecessarily complicated. Test whatever you release on your own, outside of production gr…

That's the direction PyPI is going: https://news.ycombinator.com/item?id=49007291#49047014
Post reply on HN