Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

github.com

Post reply on HN
Print view View original

141–150 of 215 posts

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#141

Post by seba_dos1 »

paddim8 wrote: post #134
seba_dos1 wrote: post #18

Earlier quoted context omitted.

Yeah, in my 20 years of using and developing on GNU/Linux the only binary compatibility issues I experienced that I can think of now were related to either Adobe Flash, Adobe Reader or games. Adobe stuff is of the kind that you'd prefer to not exist at all rather than have it fixed (and today you largely can pretend that it never existed already), and the situation for games has been pretty much fixed by Steam runtim…

Probably because your distro purposefully keeps software out of date because it is too fragile otherwise. I don't think that is reasonable at all for desktop use.

Arch?...

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#142

Post by Conscat »

Rochus wrote: post #60

So what we need is essentially a "libc virtualization". But Musl is only available on Linux, isn't it? Cosmopolitan ( https://github.com/jart/cosmopolitan ) goes further and is available also on Mac and Windows, and it uses e.g. SIMD and other performance related improvements. Unfortunately, one has to cut through the marketing "magic" to find the main engineering value; stripping away the "polyglot" shell-script hac…

If the APE concept isn't appealing to you, you may be interested in the work on LLVM libC. My friend recently delivered an under-appreciated lecture on the vision:

https://youtu.be/HtCMCL13Grg

tl;dw Google recognizes the need for a statically-linked modular latency sensitive portable POSIX runtime, and they are building it.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#143

Post by Conscat »

amelius wrote: post #5

Is there a tool that takes an executable, collects all the required .so files and produces either a static executable, or a package that runs everywhere?

Someone already mentioned AppImage, but I'd like to draw attention to this alternate implementation that executes as a POSIX shell script, making it possible to dynamic dispatch different programs on different architectures. e.g. a fat binary for ARM and x64.

https://github.com/mgord9518/shappimage

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#144

Post by Splizard »

leni536 wrote: post #104

Do I get this right that this effectively dlopens glibc (indirectly) into an executable that is statically linked to musl? How can the two runtimes coexist? What about malloc/free? AFAIK both libc's allocators take ownership of brk, that can't be good. What about malloc/free across the dynamic library interface? There are certainly libraries that hand out allocated objects and expect the user to free them, but that's…

You have to tell musl to use mmap instead of brk. You're right that it doesn't work in all cases but as long as you switch TLS on calls (and callbacks), at least with a project the size of Godot, you can approach a workable solution.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#145

Post by uecker »

abigail95 wrote: post #100
mgaunard wrote: post #36

It's funny how people insist on wanting to link everything statically when shared libraries were specifically designed to have a better alternative. Even worse is containers, which has the disadvantage of both.

Why would I want to be constantly calling into code I have no control over, that may or may not exist, that may or may not be tampered with. I lose control of the execution state. I have to follow the calling conventions which let my flags get clobbered. To forego all of the above including link time optimization for the benefit of what exactly? Imagine developing a C program where every object file produced during c…

You call into dynamic libraries so that you do not need to recompile and distribute new binaries to all your users whenever there is a security issue or other critical fix in any of the dependencies.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#146

Post by uecker »

flohofwoe wrote: post #62
mgaunard wrote: post #36

It's funny how people insist on wanting to link everything statically when shared libraries were specifically designed to have a better alternative. Even worse is containers, which has the disadvantage of both.

Dynamic libraries make a lot of sense as operating system interface when they guarantee a stable API and ABI (see Windows for how to do that) - the other scenarios where DLLs make sense is for plugin systems. But that's pretty much it, for anything else static linking is superior because it doesn't present an optimization barrier (especially for dead code elimination). No idea why the glibc can't provide API+ABI stab…

I do not think it is difficult compiling against versions by using a container.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#147

Post by sekh60 »

Conscat wrote: post #143
amelius wrote: post #5

Is there a tool that takes an executable, collects all the required .so files and produces either a static executable, or a package that runs everywhere?

Someone already mentioned AppImage, but I'd like to draw attention to this alternate implementation that executes as a POSIX shell script, making it possible to dynamic dispatch different programs on different architectures. e.g. a fat binary for ARM and x64. https://github.com/mgord9518/shappimage

So autotools but for execution instead of compilation?

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#148

Post by geocar »

mdavid626 wrote: post #118
geocar wrote: post #78

Earlier quoted context omitted.

> But you can't take .so files and make one "static" binary out of them. Yes you can! This is more-or-less what unexec does - https://news.ycombinator.com/item?id=21394916 For some reason nobody seems to like this sorcery, probably because it combines the worst of all worlds. But there's almost[1] nothing special about what the dynamic linker is doing to get those .so files into memory that it can't arrange them in o…

What if the library you use calls dlopen later? That’ll fail. There is no universal, working way to do it. Only some hacks which work in some special cases.

> What if the library you use calls dlopen later? That’ll fail.

Nonsense. xemacs could absolutely call dlopen.

> There is no universal, working way to do it. Only some hacks which work in some special cases.

So you say, but I remember not too long ago you weren't even aware it was possible, and you clearly didn't check one of the most prominent users of this technique, so maybe you should also explain why I or anyone else should give a fuck about what you think is a "hack"?

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#149

Post by wpollock »

electroly wrote: post #99
rlpb wrote: post #95

Earlier quoted context omitted.

Imagine a fully statically linked version of Debian. What happens when there’s a security update in a commonly used library? Am I supposed to redownload a rebuild of basically the entire distro every time this happens, or else what?

Steel-manning the idea, perhaps they would ship object files (.o/.a) and the apt-get equivalent would link the system? I believe this arrangement was common in the days before dynamic linking. You don't have to redownload everything, but you do have to relink everything.

> Steel-manning the idea, perhaps they would ship object files (.o/.a) and the apt-get equivalent would link the system? I believe this arrangement was common in the days before dynamic linking. You don't have to redownload everything, but you do have to relink everything.

This was indeed comon for Unix. The only way to tune the systems (or even change the timezone) was to edit the very few source files and run make, which compiled those files then linked them into a new binary.

Linking-only is (or was) much faster than recompiling.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#150

Post by Faelian2 »

Meneth wrote: post #9

That seems mostly useful for proprietary programs. I don't like it.

I did wrote a small open-source tool in Rust. And I too did encounter that kind of issue when I did start to build a .deb.

Honestly, it was the kind of bug that is not fun to fix, because it's really about dependency, and not some fun code issue. There is no point in making our life harder with this to gatekeep proprietary software to run on our platform.

Post reply on HN