[flagged]
What's missing to have reproducible builds on PyPI
11–18 of 18 posts
Re: What's missing to have reproducible builds on PyPI
#12Earlier quoted context omitted.
They use the word "Reproducible Builds" for linking the VCS commit to PyPI's repacked source code upload, that's why "reproduce the source code" sounds a little confusing. For projects with native bindings you can still fairly trivially solve this with SBOMs, and some Linux distributions are doing this for many years now. The compiler is a dependency that needs documenting, and they explicitly write they need means t…
> For projects with native bindings you can still fairly trivially solve this with SBOMs For bitwise identical results there are many small details that need to be taken care of, even in the pure-Python case: Ensuring that files appear in the wheel (zip) file in the same order (glob() doesn't guarantee this and you can definitely see different orders from run to run), timestamps are set to some fixed standard time. A…
I'm well aware of those issues (having worked on this for many years), but I still believe the majority of python packages won't be affected by this. :)
Most of these you would fix once on the relevant [build-system] and be done with it, no need to fix each individual python library.
Re: What's missing to have reproducible builds on PyPI
#13For python, it seems very difficult to actually do reproducible builds in a way that is useful, because python wheels have very implicit assumptions about paths and environemtns that do not happen anywhere else.
I don't know what your experience is, but I can pack a reproducible AppImage containing a Python virtual environment and it works okay.
Yeah, but then you are packaging the distribution, which is a valid and often a good choice, but then it is not a reproducible wheel anymore.
Re: What's missing to have reproducible builds on PyPI
#14These people are so far gone in their fantasy land... > So we either have to shrug and say, "don't use sdists if you want reproducible builds," What is this, I can't even... How can anyone expect to have a reproducible build's artifact if they downloaded the source ? Now, and most importantly, all this talk is about "pure" (as in Python-only) packages. This is, practically, worthless for most real Python projects bec…
I don't think they are talking about getting the artifact at the same time as the source or any such thing. They are saying, the metadata required to reproduce a build is not present in that sdist format, and there's no place to attach it. So, if you got an artifact and separately got the source, you couldn't verify it without another source of information about how to do the build itself in exactly the same way. It goes beyond pure reproducibility as well. Without that metadata to set up your environment, you may see bugs or other differences in your build that are not in the distributed artifact.
Re: What's missing to have reproducible builds on PyPI
#15These people are so far gone in their fantasy land... > So we either have to shrug and say, "don't use sdists if you want reproducible builds," What is this, I can't even... How can anyone expect to have a reproducible build's artifact if they downloaded the source ? Now, and most importantly, all this talk is about "pure" (as in Python-only) packages. This is, practically, worthless for most real Python projects bec…
They use the word "Reproducible Builds" for linking the VCS commit to PyPI's repacked source code upload, that's why "reproduce the source code" sounds a little confusing. For projects with native bindings you can still fairly trivially solve this with SBOMs, and some Linux distributions are doing this for many years now. The compiler is a dependency that needs documenting, and they explicitly write they need means t…
> fairly trivially solve this with SBOMs
Please try building with conda-build to understand the actual problems this entails. Your solution is a fantasy because of how library linkage works. Dynamically loaded libraries need to be able to find each other at runtime. In order to do that, they need to store the location (the filesystem path) to the library they want to load. How these links are resolved is beyond the scope of Python (on Linux, this is managed by ld and ldconf).
A lot of Python package maintainers don't understand this problem and don't understand how to make a portable library. Often times builds contain either an absolute path to another library, or a relative path... into nowhere, or a collection of paths... into questionable places. And there's no standard that tells the library authors how to do this the right way. As long as there's no standard, anyone can claim to have done it right and expect their users to accommodate them (which is what currently happens in Python).
About your other ideas: making a compiler a dependency is... an awful idea. Unless you can require the same compiler is used for every dependency in your project, at the minimum, you will have an uncontrolled (how are you going to make sure that the right compiler is used to build a package?) zoo of compilers and their versions in your project. In the worst case, compilers will embed their signatures in the generated binary code preventing other binaries from loading such code, if they are generated with a different compiler version (kind of like what happens if you try to load Linux drivers that weren't compiled with the same compiler that compiled the kernel).
* * *
Like I said: these people have next to no practical experience with the problem they are trying to solve. Why can't they just go some place else and apply themselves elsewhere is beyond my comprehension.
Re: What's missing to have reproducible builds on PyPI
#16These people are so far gone in their fantasy land... > So we either have to shrug and say, "don't use sdists if you want reproducible builds," What is this, I can't even... How can anyone expect to have a reproducible build's artifact if they downloaded the source ? Now, and most importantly, all this talk is about "pure" (as in Python-only) packages. This is, practically, worthless for most real Python projects bec…
I don't think they are talking about getting the artifact at the same time as the source or any such thing. They are saying, the metadata required to reproduce a build is not present in that sdist format, and there's no place to attach it. So, if you got an artifact and separately got the source, you couldn't verify it without another source of information about how to do the build itself in exactly the same way. It…
OK. I see. But, even if this is the problem, the solution they are trying for is bad. What Python needs is a project definition (like what Ada has in GNAT Project Manager). PyPA repeatedly proved themselves incapable of coming up with something like this (first setup.cfg, then pyproject.toml), and I don't expect them to independently discover the solution to the problem that was discovered before Python was invented. They are not the kind of people that would be able to do that. They've been at it for some fifteen years and every time they roll out a new iteration it just gets worse.
Re: What's missing to have reproducible builds on PyPI
#17Earlier quoted context omitted.
I don't think they are talking about getting the artifact at the same time as the source or any such thing. They are saying, the metadata required to reproduce a build is not present in that sdist format, and there's no place to attach it. So, if you got an artifact and separately got the source, you couldn't verify it without another source of information about how to do the build itself in exactly the same way. It…
OK. I see. But, even if this is the problem, the solution they are trying for is bad. What Python needs is a project definition (like what Ada has in GNAT Project Manager). PyPA repeatedly proved themselves incapable of coming up with something like this (first setup.cfg, then pyproject.toml), and I don't expect them to independently discover the solution to the problem that was discovered before Python was invented.…
Yeah I agree. The Python language sucks for stuff like this. Reproducibility is hardly a priority in a language where compatibility isn't even a priority. I like Python in spite of its many warts and wish it was possible to start a new language without the mistakes and have it be as popular as Python, but I don't see that ever happening. I think they've abandoned simple logic like the classic "There should be one-- and preferably only one --obvious way to do it." Nevertheless, I appreciate people trying to make the best of it.
Re: What's missing to have reproducible builds on PyPI
#18Earlier quoted context omitted.
I don't know what your experience is, but I can pack a reproducible AppImage containing a Python virtual environment and it works okay.
Yeah, but then you are packaging the distribution, which is a valid and often a good choice, but then it is not a reproducible wheel anymore.
I'm not sure I can have reproducible AppImage (SquashFS image) without reproducible wheels?