Live data from Hacker News

The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

github.com

111–120 of 215 posts

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#111

Earlier quoted context omitted.

>Linux if you configure binfmt_misc >Windows if you disable Windows Defender >OpenBSD only older versions

>> Linux > if you configure binfmt_misc I don't think that's a requirement, it'll just fall back to the shell script bootstrap without it.

On some distros, yes. On others it'll fire up Wine for whatever reason

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#113

Earlier quoted context omitted.

Does AMD use a statically linked OpenGL?

AMD uses the dynamically linked system libGL.so, usually Mesa.

So you still need dynamic linking to load the right driver for your graphics card.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#114
post #77
post #11

Earlier quoted context omitted.

There are things like this. The things I know of and can think of off the top of my head are: 1. appimage https://appimage.org/ 2. nix-bundle https://github.com/nix-community/nix-bundle 3. guix via guix pack 4. A small collection of random small projects hardly anyone uses for docker to do this (i.e. https://github.com/NilsIrl/dockerc ) 5. A docker image (a package that runs everywhere, assuming a docker runtime is a…

It should be noted that AppImages tend to be noticeably slower at runtime than other packaging methods and also very big for typical systems which include most libraries. They're good as a "compile once, run everywhere" approach but you're really accommodating edge cases here. A "works in most cases" build should also be available for that that it would benefit. And if you can, why not provide specialized packages fo…

IMO one of the best features of AppImage is that it makes it easy to extract without needing external tools. It's usually pretty easy for me to look at an AppImage and write a PKGBUILD to make a native Arch package; the format already encodes what things need to be installed where, so it's only a question of whether the libraries it contains are the same versions of what I can pull in as dependencies (either from the main repos or the AUR). If they are, my job is basically already done, and if they aren't, I can either choose to include them in the package itself assuming I don't have anything conflicting (which is fine for local use even if it's not something that's usually tolerated when publishing a package) or stick with using the AppImage.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#115
post #5

Is there a tool that takes an executable, collects all the required .so files and produces either a static executable, or a package that runs everywhere?

  mkdir chroot
  cd chroot
  for lib in $(ldd ${executable} | grep -oE '/\S+'); do
    tgt="$(dirname ${lib})"
    mkdir -p .${tgt}
    cp ${lib} .${tgt}
  done
  mkdir -p .$(dirname ${executable})
  cp ${executable} .${executable}
  tar cf ../chroot-run-anywhere.tgz .

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#116
post #60

So what we need is essentially a "libc virtualization". But Musl is only available on Linux, isn't it? Cosmopolitan ( https://github.com/jart/cosmopolitan ) goes further and is available also on Mac and Windows, and it uses e.g. SIMD and other performance related improvements. Unfortunately, one has to cut through the marketing "magic" to find the main engineering value; stripping away the "polyglot" shell-script hac…

I find it amazing how much the mess that building C/C++ code has been for so many decades seems to have influenced the direction technology, the economy and even politics has been going.

Really, what would the world look like if this problem had been properly solved? Would the centralization and monetization of the Internet have followed the same path? Would Windows be so dominant? Would social media have evolved to the current status? Would we have had a chance to fight against the technofeudalism we're headed for?

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#117
post #20

Earlier quoted context omitted.

AppImage looks like what I need, thanks. I wonder though, if I package say a .so file from nVidia, is that allowed by the license?

AppImage is not what you need. It's just an executable wrapper for the archive. To make the software cross-distro, you need to compile it manually on an old distro with old glibc, make sure all the dependencies are there, and so on. https://docs.appimage.org/reference/best-practices.html#bina... There are several automation tools to make AppImages, but they won't magically allow you to compile on the latest Fedora an…

Yeah a lot of Appimage developers make assumptions about what their systems have as well (i.e. "if I depend on something that is installed by default on Ubuntu desktop then it's fine to leave out"). For example, a while ago I installed an Appimage GUI program on a headless server that I wanted to use via X11 forwarding. I ended up having to manually install a bunch of random packages (GTK stuff, fonts, etc) to get it to run. I see Appimage as basically the same as distributing Linux binaries via .tar.gz archives, except everything's in a single file.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#118
post #78

Earlier quoted context omitted.

You can "package" all .so files you need into one file, there are many tools which do this (like a zip file). But you can't take .so files and make one "static" binary out of them.

> But you can't take .so files and make one "static" binary out of them. Yes you can! This is more-or-less what unexec does - https://news.ycombinator.com/item?id=21394916 For some reason nobody seems to like this sorcery, probably because it combines the worst of all worlds. But there's almost[1] nothing special about what the dynamic linker is doing to get those .so files into memory that it can't arrange them in o…

What if the library you use calls dlopen later? That’ll fail.

There is no universal, working way to do it. Only some hacks which work in some special cases.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#119

Earlier quoted context omitted.

Well not a static binary in the sense that's commonly meant when speaking about static linking. But you can pack .so files into the executable as binary data and then dlopen the relevant memory ranges.

Yes, that's true. But I'm always a bit sceptical about such approaches. They are not universal. You still need glibc/musl to be the same on the target system. Also, if you compile againt new glibc version, but try to run on old glibc version, it might not work. These are just strange and confusing from the end users' perspective.

> But I'm always a bit sceptical about such approaches. They are not universal. You still need glibc/musl to be the same on the target system. Also, if you compile againt new glibc version, but try to run on old glibc version, it might not work.

Why would you include most of your dynamic libraries but not your libc?

You could still run into problems if you (or your libraries) want to use syscalls that weren't available on older kernels or whatever.

Re: The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

#120

Earlier quoted context omitted.

Statically linked binaries are a huge security problem, as are containers, for the same reason. Vendors are too slow to patch. When dynamically linking against shared OS libraries, Updates are far quicker and easier. And as for the size advantage, just look at a typical Golang or Haskell program. Statically linked, two-digit megabytes, larger than my libc...

This is the theory, but not the practice. In decades of using and managing many kinds of computers I have seen only a handful of dynamic libraries for whom security updates have been useful, e.g. OpenSSL. On the other hands, I have seen countless problems caused by updates of dynamic libraries that have broken various applications, not only on Linux, but even on Windows and even for Microsoft products, such as Visual…

> In decades of using and managing many kinds of computers I have seen only a handful of dynamic libraries for whom security updates have been useful, e.g. OpenSSL.

> On the other hands, I have seen countless problems caused by updates of dynamic libraries that have broken various applications,

OpenSSL is a good example of both useful and problematic updates. The number of updates that fixed a critical security problem but needed application changes to work was pretty high.

Post reply on HN