71–80 of 137 posts
i don’t have a better answer, but this convoluted mess of bash is a smell isn’t it? i live in a different part of the dev world, but could this be written to be less obtuse so it’s more obvious what’s happening? i get that a maintainer can still get malicious code in without the same rigor as an unaffiliated contributor, but surely there’s a better way than piles of “concise” (inadvertently obfuscated?) code?
> inadvertently
Earlier quoted context omitted.
The macro defined in build_to_host.m4 is probably called on the tests subdirectory, so it gets these files as a parameter. EDIT: It is called here and will do the extraction of the backdoor when run in the 'tests' subdirectory: https://salsa.debian.org/debian/xz-utils/-/blob/debian/unsta... EDIT2: So it will get the directory as a parameter, the actual file is encoded indirectly here: https://salsa.debian.org/debian/…
Thanks. Yeah, below I learned the '####Hello####' string to be present in the "bad test file" (I haven't seen it myself). I was just not expecting a "binary" file to be basically a text file and thought the `grep` was matching post extraction somehow. That's the root of my confusion. I do understand now where the file gets localized. IIRC only the "binary" files where added secretly, right? But the build script was t…
Yes, but it is important to note that these malicious m4 scripts were only present in the tar file. They were not checked into the git repo, which is why distros that actually built from git were not affected.
Totally agree with the problem of cryptic scripts in the build process, but unfortunately, if you maintain a project that needs to support a ton of different platforms, you don't have that much choice in your build tools. Pretty much everyone agrees that the 'autoconf soup' and its tooling (essentially m4, perl, shell, make) are all horrible from a readability perspective, and the amount of people who know these tools and can review changes is getting smaller, but switching to a more modern build system often times means dropping support for some platforms.
i don’t have a better answer, but this convoluted mess of bash is a smell isn’t it? i live in a different part of the dev world, but could this be written to be less obtuse so it’s more obvious what’s happening? i get that a maintainer can still get malicious code in without the same rigor as an unaffiliated contributor, but surely there’s a better way than piles of “concise” (inadvertently obfuscated?) code?
It's a very old smell, basically.
The central problem is that back in the 80s and 90s, there were scads of different Unix-like systems, each with their own warts and missing features. And software authors wanted to minimize their build dependencies.
So many communities standardized on automating builds using shell scripts, which worked everywhere. But shell scripts were a pain to write, so people would generate shell scripts using tools like the M4 macro preprocessor.
And this is why many projects have a giant mass of opaque shell scripts, just in case someone wants to run the code on AIX or some broken ancient Unix.
If you wanted to get rid of these impenetrable thickets of shell, you could:
1. Sharply limit the number of platforms you support.
2. You could standardize on much cleaner build tools.
3. You could build more key infrastructure in languages which don't require shell to build portably.
But this would be a massive undertaking, and a ton of key C libraries are maintained by one or two unpaid volunteers. And dropping support for "Obscurnix-1997" tends to be a fairly controversial decision.
So much of our key infrastructure remains surrounded by a morass of mysterious machine-generated shell scripts.
Earlier quoted context omitted.
In addition… if your build system has things like this as OK: > xz -dc $top_srcdir/tests/files/$p | eval $i | LC_ALL=C sed "s/\(.\)/\1\n/g" | LC_ALL=C awk 'BEGIN{FS="\n";RS="\n";ORS="";m=256;for(i=0;i You should probably expect the potential for abuse? We’re moving towards complexity that is outpacing human ability for any one person to understand, explain, and thus check an entire object. And for what? Build efficie…
I’m not sure why you’d say that we’re “moving towards” this sort of build system complexity. This is 1990s autoconf bs that has not yet been excised from the Linux ecosystem. Every modern build system, even the really obtuse ones, are less insane than autoconf. And the original purpose of this was not for efficiency, but to support a huge variety of target OSes/distros/architectures, most of which are no longer used…
Never allow complexity in code or so-called engineers who ask to merge tons of shitty code. Get rid of that shit and don't trust committers blindly. Anyone who enables this crap is also a liability.
This is in general a problem with traditional permission models. Pure capabilities would never leave binaries writable by anything other than the compiler/linker, shrinking the attack surface.
can we start considering binary files committed to a repo, even as data for tests, to be a huge red flag, and that the binary files themselves should instead, to the greatest extent possible, be generated at testing time by source code that's stated as reviewable cleartext (though I think this might be very difficult for some situations). This would make it much harder (though of course we can never really say "impos…
A lot of them malformed (or output is slightly different than standard output), because they need to ensure they can work even with files generated by other programs. Bugs like ' I tried to load this file and it failed, but works in XYZ' are extremly common.
These formats are often very complex and trying things like'zeroing out a high bit' doesn't cut it. Youvwould end up with binary code encoded in source.
Edit: one of simple improvements github/other forges could do is show content of archives in a diff. The payload was hidden in a archive test file and it would be displayed in a diff instead of "binary file change, no idea what is in it"
Earlier quoted context omitted.
I see, my point was more than this shouldn’t be allowed. I think part of the problem with a lot of things is we’re allowing complexity for the sake of complexity. No one has simplicity-required checks. My previous post should say “allows things like this”.
But what are you suggesting exactly? The code fragment you quoted was awk code. Awk is a generic programming language. Any programming language can be written to be complex and unreadable.
Earlier quoted context omitted.
To be clear: the build system did not use the code fragment you quoted. This complex awk code is a later stage of the backdoor.
I see, my point was more than this shouldn’t be allowed. I think part of the problem with a lot of things is we’re allowing complexity for the sake of complexity. No one has simplicity-required checks. My previous post should say “allows things like this”.
Earlier quoted context omitted.
Thanks. Yeah, below I learned the '####Hello####' string to be present in the "bad test file" (I haven't seen it myself). I was just not expecting a "binary" file to be basically a text file and thought the `grep` was matching post extraction somehow. That's the root of my confusion. I do understand now where the file gets localized. IIRC only the "binary" files where added secretly, right? But the build script was t…
> IIRC only the "binary" files where added secretly, right? But the build script was there for people to inspect? Yes, but it is important to note that these malicious m4 scripts were only present in the tar file. They were not checked into the git repo, which is why distros that actually built from git were not affected. Totally agree with the problem of cryptic scripts in the build process, but unfortunately, if yo…
Looks like I got it backwards then. I thought, the test-files where the sneaky addition. Guess nobody cared for them...
> if you maintain a project that needs to support a ton of different platforms, you don't have that much choice in your build tools
Yeah, but, if possible, we could start porting those things into better frameworks instead of adding new features to this problematic Linux legacy code base. And maybe we could also retro-fix some of it with a better meta-layer, which generates the problematic code verbosely and standardized. If it can be done for JS a thousand times, it can be done for the *nix ecosystem once.
Lastly, part of it is cultural, too. Some people seem to get a kick out of reduced, arcane code, instead of expressive "prose". See, my example above... why the fuck is dot a shortcut for `source`?! Btw. I stumbled into this in Docker documentation[1]:
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
https://download.docker.com/linux/debian \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
$(. /tmp/os-release && echo "$VERSION_CODENAME") | sudo tee ...
sudo backdoor
VERSION_CODENAME=bookworm
Normalizing shit like this is just bad practice.