Live data from Hacker News

Path Isn't Real on Linux

blog.danielh.cc

41–50 of 120 posts

Re: Path Isn't Real on Linux

#41

Earlier quoted context omitted.

Ignorance leading to assumptions. Their eureka moment: "The shell, not the Linux kernel, is responsible for searching for executables in PATH!" makes it obvious they haven't read up on operating systems. Shame because you should know how the machine works to understand what is happening in your computer. I always recommend reading Operating Systems: Three Easy Pieces. https://pages.cs.wisc.edu/~remzi/OSTEP/

The thing is, though, that PATH being a userspace concept is a contingent detail, an accident of history, not something inherent to the concept of an operating system. You can imagine a kernel that does path searches. Why not? There's a difference between something being a certain way because it has to be that way in order to implement the semantics of the system (e.g. interrupt handlers being a privilege transition)…

* in Linux, the kernel is responsible for accepting a list of execve(2) argument-words

Yes it does, but the more surprising thing is (coming from AmigaOS with its dos.library function ReadArgs()) that the shell does this. The shell is also responsible for argument expansion - madness!

On AmigaOS, when you type "delete foo#? force", the shell passes the entire command line to the delete command. The delete command calls ReadArgs() with a template (FILE/M/A, ALL/S, QUlET/S, FORCE/S), and the standard OS function parses it into lists of files, flags, keyword arguments, etc. The "file" passed is "foo#?", and the command uses MatchFirst()/MatchNext() to do file pattern matching.

Every command (that uses ReadArgs() and didn't plump for "standard C" parsing) has the same behaviour: running the command with "?" gives you the template, which tells you how to use it. Args are parsed consistently across all programs.

Then you get "standard C", which because K&R and main(), ignores this standard Amiga parsing function and just does naive splits. Across multiple Amiga C compilers, quoting rules are inconsistent. Amiga C compilers have to produce an executable, and it knows it'll be called with a full command line, so the executable itself has to break that into words before it can call main(), and it's up to each compiler writer how they're going to do that. Urgh.

In unix-land, it's up to the shell to parse the command line, and pass only the words... hence why the shell naturally does all the filename globbing, and why you have gotchas like when these two commands are sometimes the same and sometimes they're not:

    find . -name foo*
    find . -name 'foo*'
Then we have Windows, which is like Amiga C programs - it's being passed a full command string and will have its C runtime parse it for main() to consume. There's a vague expectation that it'll do quoting "like COMMAND", which itself has very odd quoting rules. At least, most people are all using the same C compiler on Windows, so it's mostly only MSVCRT's implementation so it's mostly consistent.

Re: Path Isn't Real on Linux

#43
post #3

Why would the author think that the PATH environment variable is being used by the kernel? What an odd assumption.

Well, execve(2) and execvp(3) are both "system" functions. C (which is already black magic for some people) invokes both by calling into functions exported from libc. If you're not super dorky^Wfamiliar with low-level systems stuff, you might guess that the two functions are implemented in the same place and in the same way. That the latter is just a libc wrapper around the former that does a PATH search is arcane de…

But the man page section tells you which one is is a a kernel syscall (2) and which is a C library function (3)...

Re: Path Isn't Real on Linux

#44
post #26

Earlier quoted context omitted.

If I have a file on machineA with uid10001 and I copy the file to machineB, I might want it to retain that uid, but it shouldn't matter to machineB that it doesn't map to a real user.

You’ll see this observation all the time building containers.

Don't if you only run them with root user.

Re: Path Isn't Real on Linux

#45
post #25

Fun fact: if you've ever had bash (or another shell) complain that a file doesn't exist, even though it's on $PATH, check if it's been cached by `hash`. If the file is moved elsewhere on $PATH and bash has the old path cached, you will get an ENOENT. The entire cache can be invalidated with `hash -r`.

Holy shit

Re: Path Isn't Real on Linux

#47
post #25

Fun fact: if you've ever had bash (or another shell) complain that a file doesn't exist, even though it's on $PATH, check if it's been cached by `hash`. If the file is moved elsewhere on $PATH and bash has the old path cached, you will get an ENOENT. The entire cache can be invalidated with `hash -r`.

Is this an old behavior? I would think ENOENT would invalidate the cache entry at least.

Retested with bash 5.2.37(1)-release from Debian testing, it still does this :(

Changing $PATH does wipe it at least.

Re: Path Isn't Real on Linux

#48
post #25

Fun fact: if you've ever had bash (or another shell) complain that a file doesn't exist, even though it's on $PATH, check if it's been cached by `hash`. If the file is moved elsewhere on $PATH and bash has the old path cached, you will get an ENOENT. The entire cache can be invalidated with `hash -r`.

I think bash has an alias “rehash” that does the same as hash -r too. But zsh doesn’t have it, so “hash -r” has entered my muscle memory, as it works in both shells.

Edit: wrong shell, zsh has rehash, bash does not.

Re: Path Isn't Real on Linux

#49
post #25

Fun fact: if you've ever had bash (or another shell) complain that a file doesn't exist, even though it's on $PATH, check if it's been cached by `hash`. If the file is moved elsewhere on $PATH and bash has the old path cached, you will get an ENOENT. The entire cache can be invalidated with `hash -r`.

Is this an old behavior? I would think ENOENT would invalidate the cache entry at least.

It's still a thing in bash 5.2.37.

It's just how bash works. If there's an entry in the session cache, it uses it. Since executable paths only get cached when you run a command successfully, this only happens when it gets moved from one directory in your PATH to another after you run it once, which isn't that common.

Setting PATH or calling hash -r will clear the session cache or one could run set +h which will disable it altogether.

Re: Path Isn't Real on Linux

#50
post #20

Earlier quoted context omitted.

It seems too far to go to say that because a system library holds some implementation details that the responsibility doesn't lie with the program using them. There's all sorts of complex interdependent details that make those kind of boundary distinctions difficult in many operating systems.

On Linux the main boundary between user space and kernel is quite clear: the system call layer. It is stable and well documented. https://github.com/torvalds/linux/blob/master/Documentation/... System libraries like glibc are not part of the kernel, they are just components that can be replaced. I wrote an article about it: https://www.matheusmoreira.com/articles/linux-system-calls I even asked Greg Kroah-Hartman abo…

That is indeed one of the more well defined boundaries in the system. Also worth understanding is that programs aren't generally invoking system calls directly, for example calling interrupt 0x80, glibc provides wrapper functions that invoke system calls, blurring the boundary a bit. Further blurring the boundary is the vDSO layer that intercepts some system call wrappers for more efficient access.

At issue in this article and comment thread is the boundary between the shell, environment, and Linux. This is a blurrier boundary still because the shell sets up the environment, which is passed through the kernel, and interpreted for downstream processes, generally (but not necessarily) by that shared system library.

Post reply on HN