Live data from Hacker News

The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

github.com

101–110 of 215 posts

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#101
post #87

Earlier quoted context omitted.

At the rate things are going we'll need a container virtualization layer as well, a docker for docker if you know what I mean

"All problems in computer science can be solved by another level of indirection"

ad infinitum ;-)

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#102
post #60

So what we need is essentially a "libc virtualization". But Musl is only available on Linux, isn't it? Cosmopolitan ( https://github.com/jart/cosmopolitan ) goes further and is available also on Mac and Windows, and it uses e.g. SIMD and other performance related improvements. Unfortunately, one has to cut through the marketing "magic" to find the main engineering value; stripping away the "polyglot" shell-script hac…

At the rate things are going we'll need a container virtualization layer as well, a docker for docker if you know what I mean

I'm building in this space, I take a docker inside a microvm (vm-lite) approach.

https://github.com/smol-machines/smolvm

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#103
post #87

Earlier quoted context omitted.

At the rate things are going we'll need a container virtualization layer as well, a docker for docker if you know what I mean

"All problems in computer science can be solved by another level of indirection"

"... except for the problem of too many levels of indirection."

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#104
Do I get this right that this effectively dlopens glibc (indirectly) into an executable that is statically linked to musl? How can the two runtimes coexist? What about malloc/free? AFAIK both libc's allocators take ownership of brk, that can't be good. What about malloc/free across the dynamic library interface? There are certainly libraries that hand out allocated objects and expect the user to free them, but that's probably uncommon in graphics.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#105

Earlier quoted context omitted.

Dynamic libraries have been frowned upon since their inception as being a terrible solution to a non-existent problem, generally amplifying binary sizes and harming performance. Some fun quotes of quite notable characters on the matter here: https://harmful.cat-v.org/software/dynamic-linking/ In practice, a statically linked system is often smaller than a meticulously dynamically linked one - while there are many cop…

Statically linked binaries are a huge security problem, as are containers, for the same reason. Vendors are too slow to patch. When dynamically linking against shared OS libraries, Updates are far quicker and easier. And as for the size advantage, just look at a typical Golang or Haskell program. Statically linked, two-digit megabytes, larger than my libc...

This is the theory, but not the practice.

In decades of using and managing many kinds of computers I have seen only a handful of dynamic libraries for whom security updates have been useful, e.g. OpenSSL.

On the other hands, I have seen countless problems caused by updates of dynamic libraries that have broken various applications, not only on Linux, but even on Windows and even for Microsoft products, such as Visual Studio.

I have also seen a lot of space and time wasted by the necessity of having installed in the same system, by using various hacks, a great number of versions of the same dynamic library, in order to satisfy the conflicting requirements of various applications. I have also seen systems bricked by a faulty update of glibc, if they did not have any statically-linked rescue programs.

On Windows such problems are much less frequent only because a great number of applications bundle with the them, in their own directory, the desired versions of various dynamic libraries, and Windows is happy to load those libraries. On UNIX derivatives, this usually does not work as the dynamic linker searches only standard places for libraries.

Therefore, in my opinion static linking should always be the default, especially for something like the standard C library. Dynamic linking shall be reserved for some very special libraries, where there are strong arguments that this should be beneficial, i.e. that there really exists a need to upgrade the library without upgrading the main executable.

Golang is probably an anomaly. C-based programs are rarely much bigger when statically linked than when dynamically linked. Only using "printf" is typically implemented in such a way that it links a lot into any statically-linked program, so the C standard libraries intended for embedded computers typically have some special lightweight "printf" versions, to avoid this overhead.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#106

It's funny how people insist on wanting to link everything statically when shared libraries were specifically designed to have a better alternative. Even worse is containers, which has the disadvantage of both.

Isn't the sole reason why linux sucks(sucked?) for games and other software exactly that there is a gazillion of different libraries with different versions, so you have zero assumptions about the state of the OS, which makes making sw for it such a pain?

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#107

Earlier quoted context omitted.

At the rate things are going we'll need a container virtualization layer as well, a docker for docker if you know what I mean

I'm building in this space, I take a docker inside a microvm (vm-lite) approach. https://github.com/smol-machines/smolvm

And the cycle continues

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#108
post #5

Is there a tool that takes an executable, collects all the required .so files and produces either a static executable, or a package that runs everywhere?

I don't think it's as simple as "run this one thing to package it", so if the process rather than the format is what you're looking for, this won't work, but that sounds a lot like how AppImages work from the user perspective. My understanding is that an AppImage is basically a static binary paired with a small filesystem image containing the "root" for the application (including the expected libraries under /usr/lib or wherever they belong). I don't line everything about the format, but overall it feels a lot less prescriptive than other "universal" packages like flatpak or snap, and the fact that you can easily extract it and pick out the pieces you want to repackage without needing any external tools (there are built-in flags on the binary like --appimage-extract) in helps a lot.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#109
post #95

Earlier quoted context omitted.

Dynamic libraries have been frowned upon since their inception as being a terrible solution to a non-existent problem, generally amplifying binary sizes and harming performance. Some fun quotes of quite notable characters on the matter here: https://harmful.cat-v.org/software/dynamic-linking/ In practice, a statically linked system is often smaller than a meticulously dynamically linked one - while there are many cop…

Imagine a fully statically linked version of Debian. What happens when there’s a security update in a commonly used library? Am I supposed to redownload a rebuild of basically the entire distro every time this happens, or else what?

Libraries already break their ABI so often that continuously rebuilding/relinking everything is inevitable.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#110

Earlier quoted context omitted.

Cosmopolitan goes one further: [binaries] that runs natively on Linux + Mac + Windows + FreeBSD + OpenBSD + NetBSD + BIOS on AMD64 and ARM64 https://justine.lol/cosmopolitan/

>Linux if you configure binfmt_misc >Windows if you disable Windows Defender >OpenBSD only older versions

>> Linux

> if you configure binfmt_misc

I don't think that's a requirement, it'll just fall back to the shell script bootstrap without it.

Post reply on HN