Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux kernel will support $ORIGIN, sort of

fzakaria.com

Post reply on HN
Print view View original

41–50 of 78 posts

Re: Linux kernel will support $ORIGIN, sort of

#41

Post by the8472 »

rcxdude wrote: post #11
charcircuit wrote: post #10

Nix has caused so much busywork to be created through the entire Linux ecosystem. Rather than fixing their own project they force work upon so many others. Instead of looking for ways Nix could work with ensuring else they force everything else to be modified to work with Nix. You will constantly seem them trying to get projects to adapt a flake.nix or to replace bash scripts with a different shebang since they hardc…

>Rather than fixing their own project they force work upon so many others. Nix has put in a ton of work to make their project work, and the assumptions in the rest of the ecosystem have been fighting every step of the way. And I would argue they've been changing it for the better, considering that if you were to just 'try to make nix work' you'd probably mainly come to the conclusion that it was impossible to make wo…

Many of those assumptions are standards such as the filesystem hierarchy (granted, that's an xkcd 927 situation), nix is a bit of a special snowflake that breaks things in subtle ways (e.g. [0][1]) and needs workarounds in various places. Those workarounds are not necessarily onerous, but they would be unnecessary if it didn't deviate.

[0] https://github.com/NixOS/nixpkgs/issues/420513 [1] https://github.com/NixOS/nixpkgs/issues/94228

Re: Linux kernel will support $ORIGIN, sort of

#42

Post by myrmidon »

mort96 wrote: post #34
myrmidon wrote: post #31

Earlier quoted context omitted.

Isn't that kinda to be expected if you want to provide dynamic loading functionality (dlopen)? Is the windows situation really all that different/better (with GetProcAddress in kernel32.dll)?

The Windows situation is way different: every process is supposed to link against kernel32.dll, that's the public interface to the kernel. In Linux, glibc is just one of many C stdlib implementations, you can have many versions of glibc on the same system, etc.

Sure, but if you want dynamic loading from your c stdlib (which is defensible IMO), and you want the behavior/implementation to match with the loader, then you need some kind of coupling somewhere no?

You could have a very slim libdlopen that is used by both loader and libc, but I don't really see how that's any improvement/much different.

Re: Linux kernel will support $ORIGIN, sort of

#43

Post by sylware »

linux ELF code is loading the ELF loader only. $ORIGIN may not be a good idea since that would add more ELF complexity to the kernel.

If we are honest with ourself, ELF is the core of the issue: for executables and dynamic libraries we _now_ know it is severely obsolete on modern hardware architectures.

I am currently using my own format, excrutiatingly simple, no loader, basically a program segment, "userland syscalls" with hardware CPU synchronization. I do wrap the executables into ELF capsules to run them transparently. So simple a small RFC will be enough.

With that, I discovered that the hard part is c++ and other similar languages which are very expensive in runtime infrastructure and linking complexity. I would need to build a mesa vulkan driver with that format, and it seems the blocker is c++ (and similar language namely with grotesque and absurd syntax complexity). Thx to valve to have removed a lot of c++, for less c++... would have been much better if plain and simple C.

Re: Linux kernel will support $ORIGIN, sort of

#44

Post by pas »

roblabla wrote: post #40
myrmidon wrote: post #31

Earlier quoted context omitted.

Isn't that kinda to be expected if you want to provide dynamic loading functionality (dlopen)? Is the windows situation really all that different/better (with GetProcAddress in kernel32.dll)?

In theory, ld.so could provide a stable interface to its dynamic loading capabilities independently of glibc. Then glibc would not have to be updated in concert with ld.so. There's no inherent reason that glibc should be the library shipping a dynamic linker - we could easily be in a world where libld was developed independently from libc. On Windows, things are less modular, so it's less of an issue. That said, ther…

still, eventually there's a need to support on one platform different ld.so/glibc pairs (even if they are API/ABI compatible)

it seems nixos could set up a wrapper that invokes the right ld.so based on the executable. though at this point they could probably edit the ELF binaries and patch the fixed path to ld.so when nix is installing the program.

yeah, it seems strange that this needs kernel support. but more eBFP extension points are usually welcome, so sure, why not?

Re: Linux kernel will support $ORIGIN, sort of

#45

Post by bobajeff »

Many people here seem to think this change will allow for people to package their Linux apps to be portable across distros/distro releases. If this is true then this a great!

On Debian-based distros it kind of sucks when the version of a package is only available in a newer release so the only way to get it is either compile it yourself or upgrade your system.

I remember trying to hack around this once by downloading dependencies from a newer release and helping the program locate them but I think I ran into a issue with the linker version being hardcoded or something and gave up.

Re: Linux kernel will support $ORIGIN, sort of

#46

Post by seanhunter »

reinitctxoffset wrote: post #30
seanhunter wrote: post #9

Earlier quoted context omitted.

That’s a weird rant which is manifestly false. On Nix the benefit you get from the incompatibility is the system can roll forward and back while it’s running so you can do big changes (eg updating or rolling back) the entire system without affecting running processes. You can also run groups of processes in isolation with different sets (or versions) of packages from the rest of the system. It’s fair to argue about w…

The privileged path is already there. `/usr/bin/env` is no different from `/bin/bash`, `/bin/sh`, or any other ELF artifact at a known place. The argument is made, the argument is spurious. It is made in the other direction regarding the driver run path, just as religious, opposite ruling from purity court. No one even knows why, the trail goes cold in a mysterious 2012 commit about Mesa, it's literally a performance…

It's no different other than it's a layer of indirection which allows bash to change in a way that would not be possible if scripts directly referenced /bin/bash or /usr/bin/bash. The only thing you can't change is the /usr/bin/env binary itself but that's not something that changes very much if at all. On my nix box, bash is at /run/current-system/sw/bin/bash and "current-system" is a symlink to a particular build in /nix/store. So /usr/bin/env is creating a layer of indirection so if I rebuild the system, the previous version can be kept around and I can roll back instantly at the boot menu and it will roll everything back to a consistent package set. Now as I say, you could achieve this in other ways eg fs snapshots etc. But to say that the change doesn't give you anything is just very obviously wrong.

And you don't need to worry- it's perfectly obvious how deep your Nix expertise is.

Re: Linux kernel will support $ORIGIN, sort of

#47

Post by inigyou »

mort96 wrote: post #27
inigyou wrote: post #24

Earlier quoted context omitted.

The kernel never looks at PATH, that's why. It's a shell construct.

... and the question was: "Why doesn't the kernel look at PATH" It's not a "shell construct" either, the standard execvp libc function looks at PATH.

A surprising number of things in Linux are transparent to the kernel as they are implemented in user space. For instance, the Linux kernel doesn't support multithreading. It only supports processes sharing an address space. Which is what most people call threads - but in the kernel, they are just processes that happen to share an address space.

Re: Linux kernel will support $ORIGIN, sort of

#48

Post by roblabla »

pas wrote: post #44
roblabla wrote: post #40

Earlier quoted context omitted.

In theory, ld.so could provide a stable interface to its dynamic loading capabilities independently of glibc. Then glibc would not have to be updated in concert with ld.so. There's no inherent reason that glibc should be the library shipping a dynamic linker - we could easily be in a world where libld was developed independently from libc. On Windows, things are less modular, so it's less of an issue. That said, ther…

still, eventually there's a need to support on one platform different ld.so/glibc pairs (even if they are API/ABI compatible) it seems nixos could set up a wrapper that invokes the right ld.so based on the executable. though at this point they could probably edit the ELF binaries and patch the fixed path to ld.so when nix is installing the program. yeah, it seems strange that this needs kernel support. but more eBFP…

> though at this point they could probably edit the ELF binaries and patch the fixed path to ld.so when nix is installing the program.

That's exactly what they're doing right now - though instead of being "when installing the program", it's "when compiling the program". The problem with hardcoding the path is that it pins the "nix store" (where nix installs all of its programs) to a hardcoded location. If you want to move it, you have to rebuild all your packages - which is suboptimal.

In theory, nix could have a system in place to just patch all binaries when moving the nix store, but that would be incompatible with content-addressed derivation and otherwise break some other nice properties of the nix store.

Re: Linux kernel will support $ORIGIN, sort of

#49

Post by HexDecOctBin »

VorpalWay wrote: post #26
HexDecOctBin wrote: post #15

Earlier quoted context omitted.

Unfortunately, Glibc and loader are linked together intrinsically in the Linux ecosystem. So, if you want to be able to launch a program reliably, shipping your own loader and libc might actually be the only way.

Glibc is backward compatible though, they even have symbol versioning to provide multiple versions of the same symbol. So as long as you have the same or a newer version of glibc (than what was built against) you should be good to go. And I don't remember hearing about breakages for this. Other libraries on the system is far more hit and miss, but glibc is quite compatible. The other way around is harder though, you…

Glibc is the opposite of backwards compatible. This thread I was a part of explains some issues faced in the past: https://news.ycombinator.com/item?id=47029789

Re: Linux kernel will support $ORIGIN, sort of

#50

Post by VorpalWay »

HexDecOctBin wrote: post #49
VorpalWay wrote: post #26

Earlier quoted context omitted.

Glibc is backward compatible though, they even have symbol versioning to provide multiple versions of the same symbol. So as long as you have the same or a newer version of glibc (than what was built against) you should be good to go. And I don't remember hearing about breakages for this. Other libraries on the system is far more hit and miss, but glibc is quite compatible. The other way around is harder though, you…

Glibc is the opposite of backwards compatible. This thread I was a part of explains some issues faced in the past: https://news.ycombinator.com/item?id=47029789

Reading the linked bug report about executable stacks they fixed it? So they did the right thing. I'm not saying there will never be bugs (no non-trivial software is bug free), but as long as those are handled correctly that seems reasonable to me.
Post reply on HN