Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Clodl: Turn dynamically linked ELF binaries into self-contained closures

github.com

Post reply on HN
Print view View original

21–29 of 29 posts

Re: Clodl: Turn dynamically linked ELF binaries into self-contained closures

#21

Post by PaulDavisThe1st »

Wowfunhappy wrote: post #8
fao_ wrote: post #2

That is, given a shared library or a position independent executable (PIE), it returns a single, self-contained file packing all dependencies. [...] but closures do allow for deploying to other machines without concerns about missing dependencies. So. It's a statically linked binary, but worse? Or like a PAR file, but for C? I'm really not sure what the benefits are here over statically linked binaries. In fact, sinc…

Pretty please, tell me how to compile static binaries and I’ll do that forever. I always end up with certain external libraries still being required.

Step 3b is to make sure you don't use libraries that themselves cannot be fully statically linked (hello, GTK!).

It's cool that they use run-time dynamic loading to "modularize" certain features (such as image loading), but it seriously damages the benefits of static linkage.

Re: Clodl: Turn dynamically linked ELF binaries into self-contained closures

#22

Post by toast0 »

fao_ wrote: post #2

That is, given a shared library or a position independent executable (PIE), it returns a single, self-contained file packing all dependencies. [...] but closures do allow for deploying to other machines without concerns about missing dependencies. So. It's a statically linked binary, but worse? Or like a PAR file, but for C? I'm really not sure what the benefits are here over statically linked binaries. In fact, sinc…

It can be tricky to get static executables out of a complex build process that wasn't put together with that goal in mind.

For reasons, I was trying to do that for a while, but it was easier for me to bundle a dynamic exe with the dependencies (including an appropriate dynamic linker) rather than continue to fight with complex makefiles. Without looking at the code, it seems like this ended up similarly.

Re: Clodl: Turn dynamically linked ELF binaries into self-contained closures

#23

Post by nh2 »

PaulDavisThe1st wrote: post #21
Wowfunhappy wrote: post #8

Earlier quoted context omitted.

Pretty please, tell me how to compile static binaries and I’ll do that forever. I always end up with certain external libraries still being required.

Step 3b is to make sure you don't use libraries that themselves cannot be fully statically linked (hello, GTK!). It's cool that they use run-time dynamic loading to "modularize" certain features (such as image loading), but it seriously damages the benefits of static linkage.

GTK can be statically linked.

Example executable:

https://github.com/nh2/static-haskell-nix/releases/tag/c-sta...

It lost this ability temporarily when switching to Meson, but I fixed it in GTK3 and GTK4. But I just checked and apparently it is broken again:

https://gitlab.gnome.org/GNOME/gtk/-/issues/3774#note_109746...

Re: Clodl: Turn dynamically linked ELF binaries into self-contained closures

#24

Post by nh2 »

fao_ wrote: post #2

That is, given a shared library or a position independent executable (PIE), it returns a single, self-contained file packing all dependencies. [...] but closures do allow for deploying to other machines without concerns about missing dependencies. So. It's a statically linked binary, but worse? Or like a PAR file, but for C? I'm really not sure what the benefits are here over statically linked binaries. In fact, sinc…

I often work on static linking with Nix.

Creating statically linked end user programs that aren't minimal CLI utilities is not trivial.

You need static linking working for all your dependencies (`.a` files) of all your recursive dependencies (hundreds of C libraries, and often up to hundreds of libraries in the higher-level language you may be using).

You need to either use a very configurable package distribution like nixpkgs, or a distribution that specialises in static Linking like Alpine Linux.

These toolchains now make static linking possible and much easier, and nixpkgs can build hundreds of C and Haskell programs statically. Go is also easy. When you can statically link, it is often better, but there are still many programs for which it doesn't work yet, and just "packing up all the .so files" as proposed in the announcement is certainly easier.

For people who want to contribute to static linking, consider contributing to nixpkgs/NixOS and Alpine Linux.

Re: Clodl: Turn dynamically linked ELF binaries into self-contained closures

#25

Post by tadfisher »

PaulDavisThe1st wrote: post #19
ris wrote: post #15

Nix can do this for basically anything - even python scripts with all their dependency modules, binary extensions and all dynamic lib dependencies, packed up into a single .nar file. nix-shell can even be used as a shebang-line for scripts: https://nixos.org/manual/nix/stable/#use-as-a-interpreter

On the other hand, Nix is also the only Linux distribution to have broken ("changed") the LD_LIBRARY_PATH behavior of the runtime linker, which means 3rd party "self-contained" packages can't work there, even though they do on every other Linux distro. So there's that.

The NixOS solution to that is `buildFHSUserEnv` (soon to use bubblewrap under the hood) but obviously that's not ideal.

`nix bundle` (coming with the upcoming flakes feature) promises to make Nix-built binaries runnable without nix, for the opposite of what you want.

Re: Clodl: Turn dynamically linked ELF binaries into self-contained closures

#26

Post by jart »

ris wrote: post #15

Nix can do this for basically anything - even python scripts with all their dependency modules, binary extensions and all dynamic lib dependencies, packed up into a single .nar file. nix-shell can even be used as a shebang-line for scripts: https://nixos.org/manual/nix/stable/#use-as-a-interpreter

How useful is that though, for liberating and expropriating binaries? That things got to be at least several megabytes. If I used that as an interpreter or shell script prefix hack then I'd feel like an ant riding on the JVM's coattails.

That's one of the reasons I wrote APE. The bootloader shell script prefix is only 12kb so you can write a small program and it'll feel like it's actually your program. When you distribute it to your friends (respecting their time not asking them to become build system experts, just to try the cool thing you've written) you'll find that APE lets it run on seven operating systems too and most importantly, it'll just feel like something you've written, and that is yours, rather than, "hey grandma install oracle java first, here's how to remove the toolbar" kind of things where you're just evangelizing someone else's platform.

Re: Clodl: Turn dynamically linked ELF binaries into self-contained closures

#27

Post by PaulDavisThe1st »

nh2 wrote: post #23
PaulDavisThe1st wrote: post #21

Earlier quoted context omitted.

Step 3b is to make sure you don't use libraries that themselves cannot be fully statically linked (hello, GTK!). It's cool that they use run-time dynamic loading to "modularize" certain features (such as image loading), but it seriously damages the benefits of static linkage.

GTK can be statically linked. Example executable: https://github.com/nh2/static-haskell-nix/releases/tag/c-sta... It lost this ability temporarily when switching to Meson, but I fixed it in GTK3 and GTK4. But I just checked and apparently it is broken again: https://gitlab.gnome.org/GNOME/gtk/-/issues/3774#note_109746...

Oh, that's right. There's some option to build all modules as static libs or include them in the main .a ... sorry, I forgot about that.

Did GTK3 or GTK4 ever become relocatable on Linux?

Re: Clodl: Turn dynamically linked ELF binaries into self-contained closures

#28

Post by fao_ »

pengaru wrote: post #10
fao_ wrote: post #6

Earlier quoted context omitted.

> when have you last tried to create a statically-linked binary that used glibc, and how did that go for you? I don't write code that requires Glib, but last I remember, there are explicit (lesser-known) flags that give complete static independence :P My implicit point is that essentially, there is a slew of approaches that seem to be reinventing the wheel to solve the obvious problems in dynamic linking, but they se…

> I don't write code that requires Glib, but last I remember, there are explicit (lesser-known) flags that give complete static independence :P Glib != glibc

I actually wrote that in, but removed it accidentally when I was adding capitalisation :)

I think it's pretty clear based on context that I'm talking about GlibC though :P

Re: Clodl: Turn dynamically linked ELF binaries into self-contained closures

#29

Post by nh2 »

PaulDavisThe1st wrote: post #27
nh2 wrote: post #23

Earlier quoted context omitted.

GTK can be statically linked. Example executable: https://github.com/nh2/static-haskell-nix/releases/tag/c-sta... It lost this ability temporarily when switching to Meson, but I fixed it in GTK3 and GTK4. But I just checked and apparently it is broken again: https://gitlab.gnome.org/GNOME/gtk/-/issues/3774#note_109746...

Oh, that's right. There's some option to build all modules as static libs or include them in the main .a ... sorry, I forgot about that. Did GTK3 or GTK4 ever become relocatable on Linux?

Well, looks like you just turned right:

https://gitlab.gnome.org/GNOME/gtk/-/merge_requests/3519#not...

Post reply on HN