Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

github.com

Post reply on HN
Print view View original

41–50 of 215 posts

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#41

Post by Xraider72 »

flohofwoe wrote: post #17
iberator wrote: post #15

Earlier quoted context omitted.

Yup. Just compile it as static executable. Static binaries are very undervalued imo.

The "just" is doing a lot of heavylifting here (as detailed in the article), especially for anything that's not a trivial cmdline tool.

In my experience it seems to be an issue caused by optimizations in legacy code that relied on dlopen to implement a plugin system, or help with startup, since you could lazy load said plugins on demand and start faster.

If you forego the requirement of a runtime plugin system, is there anything realistically preventing greenfield projects from just being fully statically linked, assuming their dependencies dont rely on dlopen ?

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#42

Post by fc417fc802 »

mgaunard wrote: post #36

It's funny how people insist on wanting to link everything statically when shared libraries were specifically designed to have a better alternative. Even worse is containers, which has the disadvantage of both.

Dynamic linking exists to make a specific set of tradeoffs. Neither better nor worse than static linking in the general sense.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#43

Post by direwolf20 »

amelius wrote: post #20
TheDong wrote: post #11

Earlier quoted context omitted.

There are things like this. The things I know of and can think of off the top of my head are: 1. appimage https://appimage.org/ 2. nix-bundle https://github.com/nix-community/nix-bundle 3. guix via guix pack 4. A small collection of random small projects hardly anyone uses for docker to do this (i.e. https://github.com/NilsIrl/dockerc ) 5. A docker image (a package that runs everywhere, assuming a docker runtime is a…

AppImage looks like what I need, thanks. I wonder though, if I package say a .so file from nVidia, is that allowed by the license?

No, that's a copyright violation, and it won't run on AMD or Intel GPUs, or kernels with a different Nvidia driver version.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#44

Post by flohofwoe »

Xraider72 wrote: post #41
flohofwoe wrote: post #17

Earlier quoted context omitted.

The "just" is doing a lot of heavylifting here (as detailed in the article), especially for anything that's not a trivial cmdline tool.

In my experience it seems to be an issue caused by optimizations in legacy code that relied on dlopen to implement a plugin system, or help with startup, since you could lazy load said plugins on demand and start faster. If you forego the requirement of a runtime plugin system, is there anything realistically preventing greenfield projects from just being fully statically linked, assuming their dependencies dont rely…

It becomes tricky when you need to use system DLLs like X11 or GL/Vulkan (so you need to use the 'hacks' described in the article to work around that) - the problem is that those system DLLs then bring a dynamically linked glibc into the process, so suddenly you have two C stdlibs running side by side and the question is whether this works just fine or causes subtle breakage under the hood (e.g. the reason why MUSL doesn't implement dlopen).

E.g. in my experience: command line tools are fine to link statically with MUSL, but as soon as you need a window and 3D rendering it's not worth the hassle.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#46

Post by fc417fc802 »

mdavid626 wrote: post #7
amelius wrote: post #5

Is there a tool that takes an executable, collects all the required .so files and produces either a static executable, or a package that runs everywhere?

You can "package" all .so files you need into one file, there are many tools which do this (like a zip file). But you can't take .so files and make one "static" binary out of them.

Well not a static binary in the sense that's commonly meant when speaking about static linking. But you can pack .so files into the executable as binary data and then dlopen the relevant memory ranges.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#47

Post by arghwhat »

mgaunard wrote: post #36

It's funny how people insist on wanting to link everything statically when shared libraries were specifically designed to have a better alternative. Even worse is containers, which has the disadvantage of both.

Dynamic libraries have been frowned upon since their inception as being a terrible solution to a non-existent problem, generally amplifying binary sizes and harming performance. Some fun quotes of quite notable characters on the matter here: https://harmful.cat-v.org/software/dynamic-linking/

In practice, a statically linked system is often smaller than a meticulously dynamically linked one - while there are many copies of common routines, programs only contain tightly packed, specifically optimized and sometimes inlined versions of the symbols they use. The space and performance gain per program is quite significant.

Modern apps and containers are another issue entirely - linking doesn't help if your issue is gigabytes of graphical assets or using a container base image that includes the entire world.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#48

Post by RicoElectrico »

mgaunard wrote: post #36

It's funny how people insist on wanting to link everything statically when shared libraries were specifically designed to have a better alternative. Even worse is containers, which has the disadvantage of both.

That would be a good point if said shared libraries did not break binary backwards compatibility and behaved more like winapi.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#50

Post by account42 »

VadimPR wrote: post #31
gilli wrote: post #29

Earlier quoted context omitted.

I wish AppImage was slightly more user friendly and did not require the user to specifically make it executable.

We fix this issue by distributing ours in a tar file with the executable bit set. Linux novices can just double click on the tar to exact it and double click again on the actual appimage. Been doing it this way for years now, so it's well battle tested.

That kind of defeats the point of an AppImage though - you could just as well have a tar archive with a c classic collection of binaries + optional launcher script.
Post reply on HN