can we start considering binary files committed to a repo, even as data for tests, to be a huge red flag, and that the binary files themselves should instead, to the greatest extent possible, be generated at testing time by source code that's stated as reviewable cleartext (though I think this might be very difficult for some situations). This would make it much harder (though of course we can never really say "impossible") to embed a substantial payload in this way.
when binary files are part of a test suite, they are typically trying to illustrate some element of the program being tested, in this case a file that was incorrectly xz-encoded. Binary files like these weren't typed by hand, they will always ultimately come from something plaintext source, modulo whatever "real world" data came in, like randomly generated numbers, audio or visual data, etc.
Here's an example! My own SQLAlchemy repository has a few binary files in it! https://github.com/sqlalchemy/sqlalchemy/blob/main/test/bina... oh noes. Why are those files there? well in this case I just wanted to test that I can send large binary BLOBs into the database driver and I was lazy. This is actually pretty dumb, the two binary files here add 35K of useless crap to the source, and I could just as easily generate this binary data on the fly using a two liner that spits out random bytes. Anyone could see that two liner and know that it isn't embedding a malicious payload.
If I wanted to generate a poorly formed .xz file, I'd illustrate source code that generates random data, runs it through .xz, then applies "corruption" to it, like zeroing out the high bit of every byte. The process by which this occurs would be all reviewable in source code.
Where I might be totally off here is if you're an image processing library and you want to test filters on an image, and you have the "before" and "after" images, or something similar for audio information, or other scientifically-generated real world datapoints that have some known meaning. That might be difficult to generate programmatically, and I guess even if said data were valid, payloads could be applied steganographically. So I don't know! But just like nobody would ever accept a PR that has a "curl https://some_rando_url/myfile.zip" inside of it, we should not accept PRs that have non-cleartext binary data in them, or package them, without really vetting the contents of those binary files. The simple presence of a binary file in a PR can certainly be highlighted, github could put a huge red banner BINARY FILES IN THIS PR.
Downstream packagers for distros like Debian, Redhat etc. would ideally be similarly skeptical of new binary files that appear in the source downloads, and tooling can be applied to highlight the appearance of such files. Packagers would be on the hook to confirm the source of these binary files, or ensure they are deleted (even disabling tests if necessary) before the build process is performed.
