Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

github.com

Post reply on HN
Print view View original

61–70 of 215 posts

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#61

Post by seba_dos1 »

whizzter wrote: post #59
seba_dos1 wrote: post #18

Earlier quoted context omitted.

Yeah, in my 20 years of using and developing on GNU/Linux the only binary compatibility issues I experienced that I can think of now were related to either Adobe Flash, Adobe Reader or games. Adobe stuff is of the kind that you'd prefer to not exist at all rather than have it fixed (and today you largely can pretend that it never existed already), and the situation for games has been pretty much fixed by Steam runtim…

The solution to games is to load Windows games instead of Linux binaries. Basically the way for the year of the Linux desktop is to become Windows.

These days Linux binaries usually work fine, even older ones, and when they don't the reason is that they often don't get the same attention as their Windows counterparts.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#62

Post by flohofwoe »

mgaunard wrote: post #36

It's funny how people insist on wanting to link everything statically when shared libraries were specifically designed to have a better alternative. Even worse is containers, which has the disadvantage of both.

Dynamic libraries make a lot of sense as operating system interface when they guarantee a stable API and ABI (see Windows for how to do that) - the other scenarios where DLLs make sense is for plugin systems. But that's pretty much it, for anything else static linking is superior because it doesn't present an optimization barrier (especially for dead code elimination).

No idea why the glibc can't provide API+ABI stability, but on Linux it always comes down to glibc related "DLL hell" problems (e.g. not being able to run an executable that was created on a more recent Linux system on an older Linux system even when the program doesn't access any new glibc entry points - the usually adviced solution is to link with an older glibc version, but that's also not trivial, unless you use the Zig toolchain).

TL;DR: It's not static vs dynamic linking, just glibc being a an exceptionally shitty solution as operating system interface.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#63

Post by breezykoi »

account42 wrote: post #51
breezykoi wrote: post #40

Earlier quoted context omitted.

For distro-packaged FOSS, binary compatibility isn't really a problem. Distributions like Debian already resolve dependencies by building from source and keeping a coherent set of libraries. Security fixes and updates propagate naturally. Binary compatibility solutions mostly target cases where rebuilding isn't possible, typically closed source software. Freezing and bundling software dependencies ultimately creates…

There is however also the long tail of open source software that isn't packaged for your favorite distribution.

That is very true. But because it is open source, one can request for packaging, contribute a package, use a third-party repository, or build it from source when needed.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#64

Post by mkoubaa »

lizknope wrote: post #58
amelius wrote: post #5

Is there a tool that takes an executable, collects all the required .so files and produces either a static executable, or a package that runs everywhere?

15-30 years ago I managed a lot of commercial chip design EDA software that ran on Solaris and Linux. We had wrapper shell scripts for so many programs that used LD_LIBRARY_PATH and LD_PRELOAD to point to the specific versions of various libraries that each program needed. I used "ldd" which prints out the shared libraries a program uses.

Sounds painful. Better to distrib a separate bundle per platform and use RPATH

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#65

Post by flohofwoe »

pjmlp wrote: post #27
iberator wrote: post #15

Earlier quoted context omitted.

Yup. Just compile it as static executable. Static binaries are very undervalued imo.

We had a time when static binaries where pretty much the only thing we had available. Here is an idea, lets go back to pure UNIX distros using static binaries with OS IPC for any kind of application dynamism, I bet it will work out great, after all it did for several years. Got to put that RAM to use.

The thing with static linking is that it enables aggressive dead code elimination (e.g. DLL are a hard optimization barrier).

Even with multiple processes sharing the same DLL I would be surprised if the alternative of those processes only containing the code they actually need would increase RAM usage dramatically, especially since most processes that run in the background on a typical Linux system wouldn't event even need to go through glibc but could talk directly to the syscall interface.

DLLs are fine as operating system interface as long as they are stable (e.g. Windows does it right, glibc doesn't). But apart from operating system interfaces and plugins, overusing dynamic linking just doesn't make a lot of sense (like on most Linux systems with their package managers).

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#66

Post by pjmlp »

account42 wrote: post #56
pjmlp wrote: post #27

Earlier quoted context omitted.

We had a time when static binaries where pretty much the only thing we had available. Here is an idea, lets go back to pure UNIX distros using static binaries with OS IPC for any kind of application dynamism, I bet it will work out great, after all it did for several years. Got to put that RAM to use.

I don't think dynamic libraries fail at "utilizing" any available RAM.

Think of any program that uses dynamic libraries as extension mechanism, and now replace it with standard UNIX processes, each using any form of UNIX IPC to talk with the host process instead.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#67

Post by koffiezet »

breezykoi wrote: post #40
juliangmp wrote: post #34

Earlier quoted context omitted.

Why? Foss software also benefits from less dependency hell.

For distro-packaged FOSS, binary compatibility isn't really a problem. Distributions like Debian already resolve dependencies by building from source and keeping a coherent set of libraries. Security fixes and updates propagate naturally. Binary compatibility solutions mostly target cases where rebuilding isn't possible, typically closed source software. Freezing and bundling software dependencies ultimately creates…

It however shifts a lot of the complexity of building the application to the distro maintainer, or a software maintainer has to prioritize for which distribution they choose to build and maintain a package, because supporting them all is a nightmare and an ever shifting moving target. And it's not just a distribution problem, it's even a distribution version/release problem.

Look at the hoops you sometimes have to jump through or hacks you have to apply to make something work on Nix, just because there is no standardization or build processes assume library locations etc. And if you then raise an issue with the software maintainer - the response is often "but we don't support Nix". And if they're not Nix/Nixos users, can you blame them?

If you've ever had to compile a modern/recent software package for an old distro (I've had to do this for old RH distro's on servers which due to regulations could not be upgraded) - you're in a world of pain. And both distro and software maintainers will say "not my problem, we don't support this" - and I fully understand their stance on that, because it is far from straight forward, and only serves a limited audience.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#68

Post by pjmlp »

flohofwoe wrote: post #65
pjmlp wrote: post #27

Earlier quoted context omitted.

We had a time when static binaries where pretty much the only thing we had available. Here is an idea, lets go back to pure UNIX distros using static binaries with OS IPC for any kind of application dynamism, I bet it will work out great, after all it did for several years. Got to put that RAM to use.

The thing with static linking is that it enables aggressive dead code elimination (e.g. DLL are a hard optimization barrier). Even with multiple processes sharing the same DLL I would be surprised if the alternative of those processes only containing the code they actually need would increase RAM usage dramatically, especially since most processes that run in the background on a typical Linux system wouldn't event ev…

While at the same time it prevents extending applications, the alternatives being multiple processes using OS IPC, all of them much slower and heavier on resources than an indirect call on a dynamic library.

We started there in computing history, and outside Linux where this desire to go to the past prevails, moved on to better ways including on other UNIX systems.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#70

Post by direwolf20 »

holowoodman wrote: post #57
amelius wrote: post #52

Earlier quoted context omitted.

But this ruins the entire idea of packaging software in a self-contained way, at least for a large class of programs. It makes me wonder, does the OS still take its job of hardware abstraction seriously these days?

The OS does. Nvidia doesn't.

Does Nvidia not support OpenGL?
Post reply on HN