Live data from Hacker News

The state of binary compatibility on Linux and how to address it

jangafx.com

51–60 of 145 posts

Re: The state of binary compatibility on Linux and how to address it

#51
post #32

Here's a thought: just distribute source code. ABI issues should be mostly fixed. Most computers can compile source code fast enough for the user not to notice and cache the results so that it's never a problem again. If you want optimised code, you can do a source to source optimisation then zip and minify the file. You could compile such a library to approximately native speeds without much user-end lag using moder…

the first launch of firefox would take a few hours.

let alone the first boot of the linux kernel... :)

Re: The state of binary compatibility on Linux and how to address it

#52
post #32

Here's a thought: just distribute source code. ABI issues should be mostly fixed. Most computers can compile source code fast enough for the user not to notice and cache the results so that it's never a problem again. If you want optimised code, you can do a source to source optimisation then zip and minify the file. You could compile such a library to approximately native speeds without much user-end lag using moder…

a.k.a Rust approach

Re: The state of binary compatibility on Linux and how to address it

#53

There is no distinction between system and program libraries in Linux. We used to pretend there was one before usrmigration, but that was never good to take seriously. The distro as packager model ensures that everything is mixed together in the filesystem and is actively hostile to external packaging. Vendoring dependencies or static linking improves compatibility by choosing known working versions, but decreases in…

> Even dns resolution on glibc implies dynamic linking due to nsswitch.

Because, as far as I’ve heard, it borrowed that wholesale from Sun, who desperately needed an application to show off their new dynamic linking toy. There’s no reason they couldn’t’ve done a godsdamned daemon (that potentially dynamically loaded plugins) instead, and in fact making some sort of NSS compatibility shim that does work that way (either by linking the daemon with Glibc, or more ambitiously by reimplementing the NSS module APIs on top of a different libc) has been on my potential project list for years. (Long enough that Musl apparently did a different, less-powerful NSS shim in the meantime?)

The same applies to PAM word for word.

> Mixing static linking and dlopen doesn't make much sense, as said [in an oft-cited thread on the musl mailing list].

It’s a meh argument, I think.

It’s true that there’s something of a problem where two copies of a libc can’t coexist in a process, and that entails the problem of pulling in the whole libc that’s mentioned in the thread, but that to me seems more due to a poorly drawn abstraction boundary than anything else. Witness Windows, which has little to no problem with multiple libcs in a process; you may say that’s because most of the difficult-to-share stuff is in KERNEL32 instead, and I’d say that was exactly my point.

The host app would need to pull in a full copy of the dynamic loader? Well duh, but also (again) meh. The dynamic loader is not a trivial program, but it isn’t a huge program, either, especially if we cut down SysV/GNU’s (terrible) dynamic-linking ABI a bit and also only support dlopen()ing ELFs (elves?) that have no DT_NEEDED deps (having presumably been “statically” linked themselves).

So that thread, to me, feels like it has the same fundamental problem as Drepper’s standard rant[1] against static linking in general: it mixes up the problems arising from one libc’s particular implementation with problems inherent to the task of being a libc. (Drepper’s has much more of an attitude problem, of course.)

As for why you’d actually want to dlopen from a static executable, there’s one killer app: exokernels, loading (parts of) system-provided drivers into your process for speed. You might think this an academic fever dream, except that is how talking to the GPU works. Because of that, there’s basically no way to make a statically linked Linux GUI app that makes adequate use of a modern computer’s resources. (Even on a laptop with integrated graphics, using the CPU to shuttle pixels around is patently stupid and wasteful—by which I don’t mean you should never do it, just that there should be an alternative to doing it.)

Stretching the definitions a little, the in-proc part of a GPU driver is a very very smart RPC shim, and that’s not the only useful kind: medium-smart RPC shims like KERNEL32 and dumb ones like COM proxy DLLs and the Linux kernel’s VDSO are useful to dynamically load too.

And then there are plugins for stuff that doesn’t really want to pass through a bytestream interface (at all or efficiently), like media format support plugins (avoided by ffmpeg through linking in every media format ever), audio processing plugins, and so on.

Note that all of these intentionally have a very narrow waist[2] of an interface, and when done right they don’t even require both sides to share a malloc implementation. (Not a problem on Windows where there’s malloc at home^W^W^W a shared malloc in KERNEL32; the flip side is the malloc in KERNEL32 sucks ass and they’re stuck with it.) Hell, some of them hardly require wiring together arbitrary symbols and would be OK receiving and returning well-known structs of function pointers in an init function called after dlopen.

[1] https://www.akkadia.org/drepper/no_static_linking.html

[2] https://www.oilshell.org/blog/2022/02/diagrams.html

Re: The state of binary compatibility on Linux and how to address it

#55
post #39

> More importantly, separating the dynamic linker from the C library itself would allow multiple versions of libc to coexist, eliminating a major source of compatibility issues. This is exactly how Windows handles it, which is one of the reasons Windows maintains such strong binary compatibility. You can still run decades-old Windows software today because Microsoft doesn’t force everything to be tied to a single, ev…

> Windows maintains such strong binary compatibility

The REAL reason windows maintains binary compatibility is because it is commercial and nobody ships source code.

In fact, many applications ship a whole boatload of DLLs, which I think is the commercial equivalent of static linking.

Re: The state of binary compatibility on Linux and how to address it

#56

Earlier quoted context omitted.

As long as the library is available. Neither static nor dynamic linking is looking to solve the 20 year old binaries issue, so both will have different issues. But I think it's easier for me to find a 20 year old ISO of a Red Hat/Slackware where I can simply run the statically linked binary. Dependency hell for older distros become really difficult when the older packages are not archived anywhere anymore.

It's interesting to think how a 20 year old OS plus one program is probably a smaller bundle size than many modern Electron apps ostensibly built "for cross platform compatibility". Maybe microkernels are the way.

How should a microkernel run (WASI) WASM runtimes?

Docker can run WASM runtimes, but I don't think podman or nerdctl can yet.

From https://news.ycombinator.com/item?id=38779803 :

  docker run \
    --runtime=io.containerd.wasmedge.v1 \
    --platform=wasi/wasm \
    secondstate/rust-example-hello
From https://news.ycombinator.com/item?id=41306658 :

> ostree native containers are bootable host images that can also be built and signed with a SLSA provenance attestation; https://coreos.github.io/rpm-ostree/container/ :

  rpm-ostree rebase ostree-image-signed:registry:
  rpm-ostree rebase ostree-image-signed:docker://
Native containers run on the host and can host normal containers if a container engine is installed. Compared to an electron runtime, IDK how minimal a native container with systemd and podman, and WASM runtimes, and portable GUI rendering libraries could be.

CoreOS - which was for creating minimal host images that host containers - is now Fedora Atomic is now Fedora Atomic Desktops and rpm-ostree. Silverblue, Kinoite, Sericea; and Bazzite and Secure Blue.

Secureblue has a hardened_malloc implementation.

From https://jangafx.com/insights/linux-binary-compatibility :

> To handle this correctly, each libc version would need a way to enumerate files across all other libc instances, including dynamically loaded ones, ensuring that every file is visited exactly once without forming cycles. This enumeration must also be thread-safe. Additionally, while enumeration is in progress, another libc could be dynamically loaded (e.g., via dlopen) on a separate thread, or a new file could be opened (e.g., a global constructor in a dynamically loaded library calling fopen).

FWIU, ROP Return-Oriented Programming and Gadgets approaches have implementations of things like dynamic header discovery of static and dynamic libraries at runtime; to compile more at runtime (which isn't safe, though: nothing reverifies what's mutated after loading the PE into process space, after NX tagging or not, before and after secure enclaves and LD_PRELOAD (which some go binaries don't respect, for example).

Can a microkernel do eBPF?

What about a RISC machine for WASM and WASI?

"Customasm – An assembler for custom, user-defined instruction sets" (2024) https://news.ycombinator.com/item?id=42717357

Maybe that would shrink some of these flatpaks which ship their own Electron runtimes instead of like the Gnome and KDE shared runtimes.

Python's manylinux project specifies a number of libc versions that manylinux packages portably target.

Manylinux requires a tool called auditwheel for Linux, delicate for MacOS, and delvewheel for windows;

Auditwheel > Overview: https://github.com/pypa/auditwheel#overview :

> auditwheel is a command line tool to facilitate the creation of Python wheel packages for Linux (containing pre-compiled binary extensions) that are compatible with a wide variety of Linux distributions, consistent with the PEP 600 manylinux_x_y, PEP 513 manylinux1, PEP 571 manylinux2010 and PEP 599 manylinux2014 platform tags.

> auditwheel show: shows external shared libraries that the wheel depends on (beyond the libraries included in the manylinux policies), and checks the extension modules for the use of versioned symbols that exceed the manylinux ABI.

> auditwheel repair: copies these external shared libraries into the wheel itself, and automatically modifies the appropriate RPATH entries such that these libraries will be picked up at runtime. This accomplishes a similar result as if the libraries had been statically linked without requiring changes to the build system. Packagers are advised that bundling, like static linking, may implicate copyright concerns

github/choosealicense.com: https://github.com/github/choosealicense.com

From https://news.ycombinator.com/item?id=42347468 :

> A manylinux_x_y wheel requires glibc>=x.y. A musllinux_x_y wheel requires musl libc>=x.y; per PEP 600

Re: The state of binary compatibility on Linux and how to address it

#57

Earlier quoted context omitted.

And please, statically linking everything is NOT a solution -- the only reason I can run some games from 20 years ago still on my recent Linux is because they didn't decide to stupidly statically link everything, so I at least _can_ replace the libraries with hooks that make the games work with newer versions.

As long as the library is available. Neither static nor dynamic linking is looking to solve the 20 year old binaries issue, so both will have different issues. But I think it's easier for me to find a 20 year old ISO of a Red Hat/Slackware where I can simply run the statically linked binary. Dependency hell for older distros become really difficult when the older packages are not archived anywhere anymore.

I've recently had to do this (to bisect when a change introduced a superficial bug into a 20-year-old program). I think "simply run" is viewing Linux of that era through rose-tinted glasses.

Even for simple 2D "Super VGA" you're needing to choose the correct XFree86 implementation and still tweak your Xorg configuration. The emulated hardware also has bugs, since most of the focus is now on virtio drivers.

(The 20-year-old program was linked against libsdl, which amusingly means on my modern system it supports Wayland with no issues.)

Re: The state of binary compatibility on Linux and how to address it

#58

Earlier quoted context omitted.

It's interesting to think how a 20 year old OS plus one program is probably a smaller bundle size than many modern Electron apps ostensibly built "for cross platform compatibility". Maybe microkernels are the way.

How should a microkernel run (WASI) WASM runtimes? Docker can run WASM runtimes, but I don't think podman or nerdctl can yet. From https://news.ycombinator.com/item?id=38779803 : docker run \ --runtime=io.containerd.wasmedge.v1 \ --platform=wasi/wasm \ secondstate/rust-example-hello From https://news.ycombinator.com/item?id=41306658 : > ostree native containers are bootable host images that can also be built and sign…

Return oriented programming: https://en.wikipedia.org/wiki/Return-oriented_programming

/? awesome return oriented programming sire:github.com https://www.google.com/search?q=awesome+return+oriented+prog...

This can probably find multiple versions of libc at runtime, too: https://github.com/0vercl0k/rp :

> rp++ is a fast C++ ROP gadget finder for PE/ELF/Mach-O x86/x64/ARM/ARM64 binaries.

Re: The state of binary compatibility on Linux and how to address it

#59
post #46

Earlier quoted context omitted.

I hate compiling. 9/10 something goes wrong. Sometimes I can fix it, other times I just abandon the effort and use something else. These days I just use packages or docker images and if that doesn't work out I'm moving on, ain't nobody got time for this. You really can't expect people who just want to use their computers and don't even know what a compiler is to get involved in a process like that.

No, end users need not get involved, it could/should be handled by the operating system.

Gentoo? Compiling big packages takes ages.

Re: The state of binary compatibility on Linux and how to address it

#60
post #32

Here's a thought: just distribute source code. ABI issues should be mostly fixed. Most computers can compile source code fast enough for the user not to notice and cache the results so that it's never a problem again. If you want optimised code, you can do a source to source optimisation then zip and minify the file. You could compile such a library to approximately native speeds without much user-end lag using moder…

a.k.a Rust approach

Slow and usually pointless for minor updates.
Post reply on HN