Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

github.com

Post reply on HN
Print view View original

51–60 of 215 posts

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#51

Post by account42 »

breezykoi wrote: post #40
juliangmp wrote: post #34

Earlier quoted context omitted.

Why? Foss software also benefits from less dependency hell.

For distro-packaged FOSS, binary compatibility isn't really a problem. Distributions like Debian already resolve dependencies by building from source and keeping a coherent set of libraries. Security fixes and updates propagate naturally. Binary compatibility solutions mostly target cases where rebuilding isn't possible, typically closed source software. Freezing and bundling software dependencies ultimately creates…

There is however also the long tail of open source software that isn't packaged for your favorite distribution.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#52

Post by amelius »

direwolf20 wrote: post #43
amelius wrote: post #20

Earlier quoted context omitted.

AppImage looks like what I need, thanks. I wonder though, if I package say a .so file from nVidia, is that allowed by the license?

No, that's a copyright violation, and it won't run on AMD or Intel GPUs, or kernels with a different Nvidia driver version.

But this ruins the entire idea of packaging software in a self-contained way, at least for a large class of programs.

It makes me wonder, does the OS still take its job of hardware abstraction seriously these days?

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#53

Post by alas44 »

amelius wrote: post #5

Is there a tool that takes an executable, collects all the required .so files and produces either a static executable, or a package that runs everywhere?

There is this project "actually portable executable"/cosmopolitan libc https://github.com/jart/cosmopolitan that allows a compile once execute anywhere style type of C++ binary

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#54

Post by account42 »

flohofwoe wrote: post #44
Xraider72 wrote: post #41

Earlier quoted context omitted.

In my experience it seems to be an issue caused by optimizations in legacy code that relied on dlopen to implement a plugin system, or help with startup, since you could lazy load said plugins on demand and start faster. If you forego the requirement of a runtime plugin system, is there anything realistically preventing greenfield projects from just being fully statically linked, assuming their dependencies dont rely…

It becomes tricky when you need to use system DLLs like X11 or GL/Vulkan (so you need to use the 'hacks' described in the article to work around that) - the problem is that those system DLLs then bring a dynamically linked glibc into the process, so suddenly you have two C stdlibs running side by side and the question is whether this works just fine or causes subtle breakage under the hood (e.g. the reason why MUSL d…

X11 actually has a stable wire protocol so you don't strictly need any dynamic libraries for that - it's just that no one bothers because if you want X11 then you most likely also want GPU access where you do need to load hardware-specific libraries.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#55

Post by mdavid626 »

fc417fc802 wrote: post #46
mdavid626 wrote: post #7

Earlier quoted context omitted.

You can "package" all .so files you need into one file, there are many tools which do this (like a zip file). But you can't take .so files and make one "static" binary out of them.

Well not a static binary in the sense that's commonly meant when speaking about static linking. But you can pack .so files into the executable as binary data and then dlopen the relevant memory ranges.

Yes, that's true.

But I'm always a bit sceptical about such approaches. They are not universal. You still need glibc/musl to be the same on the target system. Also, if you compile againt new glibc version, but try to run on old glibc version, it might not work.

These are just strange and confusing from the end users' perspective.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#56

Post by account42 »

pjmlp wrote: post #27
iberator wrote: post #15

Earlier quoted context omitted.

Yup. Just compile it as static executable. Static binaries are very undervalued imo.

We had a time when static binaries where pretty much the only thing we had available. Here is an idea, lets go back to pure UNIX distros using static binaries with OS IPC for any kind of application dynamism, I bet it will work out great, after all it did for several years. Got to put that RAM to use.

I don't think dynamic libraries fail at "utilizing" any available RAM.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#57

Post by holowoodman »

amelius wrote: post #52
direwolf20 wrote: post #43

Earlier quoted context omitted.

No, that's a copyright violation, and it won't run on AMD or Intel GPUs, or kernels with a different Nvidia driver version.

But this ruins the entire idea of packaging software in a self-contained way, at least for a large class of programs. It makes me wonder, does the OS still take its job of hardware abstraction seriously these days?

The OS does. Nvidia doesn't.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#58

Post by lizknope »

amelius wrote: post #5

Is there a tool that takes an executable, collects all the required .so files and produces either a static executable, or a package that runs everywhere?

15-30 years ago I managed a lot of commercial chip design EDA software that ran on Solaris and Linux. We had wrapper shell scripts for so many programs that used LD_LIBRARY_PATH and LD_PRELOAD to point to the specific versions of various libraries that each program needed. I used "ldd" which prints out the shared libraries a program uses.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#59

Post by whizzter »

seba_dos1 wrote: post #18
Meneth wrote: post #9

That seems mostly useful for proprietary programs. I don't like it.

Yeah, in my 20 years of using and developing on GNU/Linux the only binary compatibility issues I experienced that I can think of now were related to either Adobe Flash, Adobe Reader or games. Adobe stuff is of the kind that you'd prefer to not exist at all rather than have it fixed (and today you largely can pretend that it never existed already), and the situation for games has been pretty much fixed by Steam runtim…

The solution to games is to load Windows games instead of Linux binaries.

Basically the way for the year of the Linux desktop is to become Windows.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#60

Post by Rochus »

So what we need is essentially a "libc virtualization".

But Musl is only available on Linux, isn't it? Cosmopolitan (https://github.com/jart/cosmopolitan) goes further and is available also on Mac and Windows, and it uses e.g. SIMD and other performance related improvements. Unfortunately, one has to cut through the marketing "magic" to find the main engineering value; stripping away the "polyglot" shell-script hacks and the "Actually Portable Executable" container (which are undoubtedly innovative), the core benefit proposition of Cosmopolitan is indeed a platform-agnostic, statically-linked C standard (plus some Posix) library that performs runtime system call translation, so to say "the Musl we have been waiting for".

Post reply on HN