Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A surprise with how '#!' handles its program argument in practice

utcc.utoronto.ca

Post reply on HN
Print view View original

91–100 of 116 posts

Re: A surprise with how '#!' handles its program argument in practice

#91

Post by Arch-TK »

jolmg wrote: post #49
Arch-TK wrote: post #48

Earlier quoted context omitted.

/bin is the "standard" location for bash on a subset of Linux distributions and basically no other Unix... So it's not really a standard. /bin/sh is a much more common convention but once again, not a standard. There really isn't a truly portable shebang, but the same can be said about executables themselves. As part of the build or install step of whatever thing you're making, you should really be looking these up a…

Sorry, I should probably think more widely, but I was just considering Linux distros. > /bin is the "standard" location for bash on a subset of Linux distributions Considering "location" such that it includes /bin symlinks, that would be nearly all distros, I would think... > What's more, bash isn't a standard shell. De facto and specifically among Linux distros, it is. It's probably an underestimate that 95% of all…

It's only really NixOS as far as I know that doesn't ever put bash in /bin/bash (as far as Linux distributions go). But, on the other hand, there are quite a few distros (or at least flavours of distros) which don't ship bash by default (alpine, minimal versions of most distros, and embedded-Linux focused stuff if you count it). I imagine the most common "installation" of Linux is userspace in a container (yeah I know there's no kernel there, but nobody who talks about "Linux" broadly speaking specifically cares about the kernel) and a good chunk of those will be minimal with no bash.

Bash has to be explicitly installed on OpenBSD, FreeBSD, NetBSD (I think, haven't used it in a while) and probably a bunch of others. And in all of those cases (that I know of) it doesn't end up in /bin/bash once installed.

The default bash shipped on macs is so abhorrently ancient that it would be strictly better if it didn't exist because it would reduce the number of people who think bash scripts I write are broken (they're not broken, they just inevitably depend on some bash 4+ feature). Moreover, hardcoding /bin/bash as your shebang in this case will prevent anyone from remediating this problem by installing a non-ancient bash because the old one in /bin/bash will still get used.

Re: A surprise with how '#!' handles its program argument in practice

#92

Post by masfuerte »

SAI_Peregrinus wrote: post #90
cm2187 wrote: post #35

Earlier quoted context omitted.

tbh it is lame for any program reading a text file to not support BOM. It's just one if.

UTF-8 is a text format with no BOM. Just like ASCII doesn't support a BOM. The BOM is a UTF-16 or UTF-32 thing, so "UTF-8 with BOM" is a binary file that happens to contain some UTF-8 strings as well. Since it's not a text file, it makes sense that utilities expecting text files don't handle it.

Eh? A utf8 file starting with ZERO WIDTH NO-BREAK SPACE is not a text file? How do you figure that?

Re: A surprise with how '#!' handles its program argument in practice

#93

Post by johnisgood »

Guvante wrote: post #83
jlokier wrote: post #78

Earlier quoted context omitted.

> /usr/bin/env and /bin/sh are part of the POSIX standard, this is why NixOS has those available. Contrary to popular belief, those aren't in the POSIX standard. The following are not in the POSIX standard, they are just widely implemented: - "#!" line. - /bin/sh as the location of a POSIX compliant shell, or any shell. - /usr/bin/env as the location of an env program. - The -S option to env.

I think you are using "not required by the POSIX standard" when you say "not in" which is not an accurate shorthand. #! is certainly in the POSIX standard as the exact topic of "is /bin/sh always a POSIX" shell is a discussion point (it is not guaranteed since there were systems that existed at the time that had a non-POSIX shell there)

Are they in POSIX? I do not think they are. All of them is a convention from what I remember.

Shebang is a kernel feature, for example, and POSIX does define the sh shell language and utilities, but does not specify how executables are invoked by the kernel.

Similarly, POSIX only requires that sh exists somewhere in the PATH, and the /bin/sh convention comes from the traditional Unix and FHS (Filesystem Hierarchy Standard), but POSIX does not mandate filesystem layout.

... and so on.

Correct me if I am wrong, perhaps with citations?

Re: A surprise with how '#!' handles its program argument in practice

#94

Post by NegativeK »

coppsilgold wrote: post #44

There is no security issue here. The file with the '#!' needs to be executable, and at that point it doesn't matter what it invokes because you made it executable. It could have shellcode in it or it could call python3 which can also execute shellcode. Or more likely, it would just be a malware binary which you deliberately gave permissions to and executed.

It's a vulnerability via pathing, not a worry that the shebang script could be malicious.

Someone may have dropped a malicious executable somewhere in the user's path that the shebang calls. The someone shouldn't be able to do that, but "shouldn't" isn't enough for security.

Or maybe the relatively pathed executable has unexpected interactions with the shebanged script, compared to what the script author expected.

Etc.

Re: A surprise with how '#!' handles its program argument in practice

#95

Post by hebelehubele »

mbreese wrote: post #85
zahlman wrote: post #75

Earlier quoted context omitted.

If you're okay with running the script only from the parent directory of the venv (I guess you have things set up so that's the project root), fine. You never have to "worry about whether the environment was activated", unless your code depends on those environment variables (in which case your shebang trick won't help you). Just specify the path to the venv's Python executable. You aren't really intended to put your…

I use venv's to keep library installs out of the global space. My python scripts won't run without those libraries, so running from the project root is pretty much required. I don't use the variables set by activating the venv, so that's not a real concern. I've done this for years to keep my individual projects separate and not changing activations when switching directories. I also make sure to only call `venv/bin/…

I also use the trick to insert new lookup paths.

  project_root = os.path.dirname(os.path.abspath(__file__))
  sys.path.insert(0, project_root)

Re: A surprise with how '#!' handles its program argument in practice

#96

Post by adastra22 »

tombert wrote: post #10
adastra22 wrote: post #9

Earlier quoted context omitted.

Is this UNIX?

This is NixOS, so no, it's Linux. I guess I just hoped it would work on Linux as well.

Linux is UNIX in the context of my question. On Linux the shebang is actually handled by the kernel. When you load a binary and attempt to execute it with a syscall, the kernel reads the first few bytes of the binary. If it is an ELF header, it executes the machine code as you would expect. If the first two bytes are "#!", then it interprets it as a shebang header and loads the specified binary to interpret it.

Again, this is kernel code. I admit I'm a bit confused as to why it didn't work on your system. This shouldn't be handled at the shell level.

Re: A surprise with how '#!' handles its program argument in practice

#97

Post by kevincox »

saintfire wrote: post #12
tombert wrote: post #2

Huh. I wish I had known this before. NixOS is annoying because everything is weird and symlinked and so I find myself fairly frequently making the mistake of writing `#!/bin/bash`, only to be told it can't find it, and I have to replace the path with `/run/current-system/sw/bin/bash`. Or at least I thought I did; apparently I can just have done `#!bash`. I just tested this, and it worked fine. You learn something new…

You can use `#!/usr/bin/env bash` on NixOS

Assuming bash is in $PATH. Which it often is.

Re: A surprise with how '#!' handles its program argument in practice

#98

Post by inetknght »

mbreese wrote: post #66

I do this all the time for python virtualenvs. Instead of calling ‘#!/usr/bin/env python3’ I will call ‘#!venv/bin/python3’. This way I don’t have to worry about whether the environment was activated or not.

This is terrible. I hope you're not adding a whole venv to version control.

What if the user doesn't have a venv created? What if they created it in a different directory? What if they created a second venv and want to use that instead? What if the user uses `.venv` instead of `venv`?

`#!/usr/bin/env python3` solves most of that.

Re: A surprise with how '#!' handles its program argument in practice

#99

Post by lazide »

b33j0r wrote: post #77
p4cmanus3r wrote: post #61

The authors previous article about "...(not) using #!/usr/bin/env whatever" doesn't sit well with me. "The only reason to start your script with '#!/usr/bin/env ' is if you expect your script to run on a system where Bash or whatever else isn't where you expect (or when it has to run on systems that have ' ' in different places, which is probably most common for third party packages)." His very first point is how you…

Never thought of it this way; isn’t it always safe to assume env is in PATH? Maybe `#! env ` could be considered a DSL for hashbangs. My reasoning is that `/usr/bin/env` is the thing that seems to be hard-coded to a system path, in most cases.

Some environments (like containers) have no, or extremely incomplete PATHs.

Re: A surprise with how '#!' handles its program argument in practice

#100

Post by teo_zero »

mbreese wrote: post #65
teo_zero wrote: post #36

Earlier quoted context omitted.

Wait... your interpreter reads from stdin. Shouldn't it read its first arg, instead?

IIRC interpreters can also read their programs from stdin. Try piping a script to bash and see if it works. (Actually, this is how the `curl install.sh | bash` anti pattern works. )

Independently of what the interpreter can do, it's #! that doesn't pass the file to it.
Post reply on HN