PyPI Blog: Releases now reject new files after 14 days
1–10 of 61 posts
Re: PyPI Blog: Releases now reject new files after 14 days
#2There seems to be a severe lack of hash pinning in "modern" software ecosystems. We figured out how to do it 20+ years ago with Git bringing hash addressed storage to the masses. Coming from a different background it was very surprising for me to see things like docker images, packages and github actions being updated at the whim of upstream registry. I much prefer the philosophy where builds are fully offline and predictable, even if not fully reproducible.
Re: PyPI Blog: Releases now reject new files after 14 days
#3I’m a bit surprised this is possible in the first place. I get that you might not be able to upload everything in one go, but it feels like you should “start” and “finish” a release in that case, and once it’s finished you can’t modify it.
I guess the use case is that you might want to build a wheel for an older release for a newer version of Python?
Re: PyPI Blog: Releases now reject new files after 14 days
#4I’m a bit surprised this is possible in the first place. I get that you might not be able to upload everything in one go, but it feels like you should “start” and “finish” a release in that case, and once it’s finished you can’t modify it. I guess the use case is that you might want to build a wheel for an older release for a newer version of Python?
The use case of uploading new wheels to an old release was actually (kind of) an accident: PyPI’s current upload API is stateless and originally there was only one file per release (sdists), so there was no need for a start-finish transition for releases.
(This will hopefully change pretty soon, with the “upload 2.0” work.)
Re: PyPI Blog: Releases now reject new files after 14 days
#5> To quantify how disruptive this change would be to existing workflows, the PyPI database was queried for projects that have published new files to old releases
While this may quantify how disruptive the change would be to those projects that are able to and do upload additional binaries to PyPI later, it fails to quantify how many projects already completely circumvent this block before it is even introduced.
e.g. If you tell pip to install from source.. the result may already be that you install a binary that PyPI never saw. A common hack for dealing with NVidia internals, which can explode into a large CUDA major version x GPU arch x platform x implementation x python_version cartesian product. The "extras" mechanism is not quite sufficient to model such combinations.
sample code: https://github.com/Dao-AILab/causal-conv1d/blob/4f6ae4e26ae5... https://pypi.org/project/causal-conv1d/
Re: PyPI Blog: Releases now reject new files after 14 days
#6There seems to be a severe lack of hash pinning in "modern" software ecosystems. We figured out how to do it 20+ years ago with Git bringing hash addressed storage to the masses. Coming from a different background it was very surprising for me to see things like docker images, packages and github actions being updated at the whim of upstream registry. I much prefer the philosophy where builds are fully offline and pr…
I’m not sure what this has to do with TFA: Python does have hash-pinning. TFA is not about modifying existing files on the index (PyPI doesn’t allow that), but about adding new files to a pre-existing release. But that doesn’t change the hash of older distributions on that release.
Re: PyPI Blog: Releases now reject new files after 14 days
#7There seems to be a severe lack of hash pinning in "modern" software ecosystems. We figured out how to do it 20+ years ago with Git bringing hash addressed storage to the masses. Coming from a different background it was very surprising for me to see things like docker images, packages and github actions being updated at the whim of upstream registry. I much prefer the philosophy where builds are fully offline and pr…
Would not count that as 20 years of sticking to that philosophy, though.
We also figured out 20 years ago that SHA1 was not quite as strong as initially estimated, and not quite 10 years ago that generating two colliding documents was merely a matter of some serious computing power. A few projects went ahead and changed the name of their master branch, but SHA256 preference remains elusive.
Re: PyPI Blog: Releases now reject new files after 14 days
#814 days still seems way too long to me. As a user I thought releases on pypi were immutable!
Re: PyPI Blog: Releases now reject new files after 14 days
#914 days still seems way too long to me. As a user I thought releases on pypi were immutable!
Files are immutable on PyPI, releases are not (because releases are comprised of a set of files, and files are uploaded one by one).
This is unintuitive, but the TL;DR is that files will never change on PyPI, but (previously) a user could upload a new file to a release years after their last upload to that release. This has some legitimate use cases (like allowing people to support new Python versions without bumping a package’s version), but also makes introduces challenges around locking and release security that are elaborated in the thread linked by the blog post.
I agree this could probably be ratcheted down from 14 days over time, though.
Re: PyPI Blog: Releases now reject new files after 14 days
#10This seems like common sense configuration management 101. If I download v1.2 and it’s been published then it should be considered released and not modifiable. With exceptions for ‘dev’ releases of course. I have never published anything on PyPI but I would expect there is a publish button and finalize (?) optional button that if not checked after 14 days makes it final ?