Live data from Hacker News

The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

github.com

71–80 of 215 posts

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#71
post #52

Earlier quoted context omitted.

No, that's a copyright violation, and it won't run on AMD or Intel GPUs, or kernels with a different Nvidia driver version.

But this ruins the entire idea of packaging software in a self-contained way, at least for a large class of programs. It makes me wonder, does the OS still take its job of hardware abstraction seriously these days?

It does, and one way it does that is by dynamically loading the right driver code for your hardware.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#72

Earlier quoted context omitted.

The OS does. Nvidia doesn't.

Does Nvidia not support OpenGL?

Not really. Nvidia-OpenGL is incompatible to all existing OS OpenGL interfaces, so you need to ship a separate libGL.so if you want to run on Nvidia. In some cases you even need separate binaries, because if you dynamically link against Nvidia's libGL.so, it won't run with any other libGL.so. Sometimes also vice versa.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#73
post #69

If you're using dlopen(), you're just reimplementing the dynamic linker.

that's cute, but dismissive, sort of like "if you use popen(), you are reimplementing bash". There is so much hair in ld nobody wants to know about — parsing elf, ctors/dtors, ...

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#74

It's funny how people insist on wanting to link everything statically when shared libraries were specifically designed to have a better alternative. Even worse is containers, which has the disadvantage of both.

Dynamic libraries have been frowned upon since their inception as being a terrible solution to a non-existent problem, generally amplifying binary sizes and harming performance. Some fun quotes of quite notable characters on the matter here: https://harmful.cat-v.org/software/dynamic-linking/ In practice, a statically linked system is often smaller than a meticulously dynamically linked one - while there are many cop…

Statically linked binaries are a huge security problem, as are containers, for the same reason. Vendors are too slow to patch.

When dynamically linking against shared OS libraries, Updates are far quicker and easier.

And as for the size advantage, just look at a typical Golang or Haskell program. Statically linked, two-digit megabytes, larger than my libc...

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#75
post #66

Earlier quoted context omitted.

I don't think dynamic libraries fail at "utilizing" any available RAM.

Think of any program that uses dynamic libraries as extension mechanism, and now replace it with standard UNIX processes, each using any form of UNIX IPC to talk with the host process instead.

In theory there might be a different RAM usage with the two approaches. In practice there is not.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#77
post #11
post #5

Is there a tool that takes an executable, collects all the required .so files and produces either a static executable, or a package that runs everywhere?

There are things like this. The things I know of and can think of off the top of my head are: 1. appimage https://appimage.org/ 2. nix-bundle https://github.com/nix-community/nix-bundle 3. guix via guix pack 4. A small collection of random small projects hardly anyone uses for docker to do this (i.e. https://github.com/NilsIrl/dockerc ) 5. A docker image (a package that runs everywhere, assuming a docker runtime is a…

It should be noted that AppImages tend to be noticeably slower at runtime than other packaging methods and also very big for typical systems which include most libraries. They're good as a "compile once, run everywhere" approach but you're really accommodating edge cases here.

A "works in most cases" build should also be available for that that it would benefit. And if you can, why not provide specialized packages for the edge cases?

Of course, don't take my advice as-is, you should always thoroughly benchmark your software on real systems and choose the tradeoffs you're willing to make.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#78
post #5

Is there a tool that takes an executable, collects all the required .so files and produces either a static executable, or a package that runs everywhere?

You can "package" all .so files you need into one file, there are many tools which do this (like a zip file). But you can't take .so files and make one "static" binary out of them.

> But you can't take .so files and make one "static" binary out of them.

Yes you can!

This is more-or-less what unexec does

- https://news.ycombinator.com/item?id=21394916

For some reason nobody seems to like this sorcery, probably because it combines the worst of all worlds.

But there's almost[1] nothing special about what the dynamic linker is doing to get those .so files into memory that it can't arrange them in one big file ahead of time!

[1]: ASLR would be one of those things...

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#79
post #60

So what we need is essentially a "libc virtualization". But Musl is only available on Linux, isn't it? Cosmopolitan ( https://github.com/jart/cosmopolitan ) goes further and is available also on Mac and Windows, and it uses e.g. SIMD and other performance related improvements. Unfortunately, one has to cut through the marketing "magic" to find the main engineering value; stripping away the "polyglot" shell-script hac…

At the rate things are going we'll need a container virtualization layer as well, a docker for docker if you know what I mean

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#80
post #39

I'd never heard of detour. That's a pretty cool hack.

they were prominent in game hacking 2005ish windows made hooking into game code much easier than before

Aren't all DLLs on the Windows platform compiled with an unusual instruction at the start of each function? This makes it possible to somehow hot patch the DLL after it is already in memory
Post reply on HN