Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A surprise with how '#!' handles its program argument in practice

utcc.utoronto.ca

Post reply on HN
Print view View original

101–110 of 116 posts

Re: A surprise with how '#!' handles its program argument in practice

#101

Post by wahern »

johnisgood wrote: post #93
Guvante wrote: post #83

Earlier quoted context omitted.

I think you are using "not required by the POSIX standard" when you say "not in" which is not an accurate shorthand. #! is certainly in the POSIX standard as the exact topic of "is /bin/sh always a POSIX" shell is a discussion point (it is not guaranteed since there were systems that existed at the time that had a non-POSIX shell there)

Are they in POSIX? I do not think they are. All of them is a convention from what I remember. Shebang is a kernel feature, for example, and POSIX does define the sh shell language and utilities, but does not specify how executables are invoked by the kernel. Similarly, POSIX only requires that sh exists somewhere in the PATH, and the /bin/sh convention comes from the traditional Unix and FHS (Filesystem Hierarchy Sta…

You're definitely correct. "#!" is reserved (see Rationale C.2.1), but not required, though it's described as "ubiquitous" (see Rationale C.1.7). "/bin/sh" isn't required either, but arguably ubiquitous in that there's always some shell located there. The proper way to find the POSIX-conformant shell is with `command -v sh` (which is equivalent to using `getconf PATH` and then searching for sh), and POSIX counsels to discover the path and substitute it inline when installing scripts (see Application Usage in sh utility specification.)

IME /bin/sh is invariably sufficiently POSIX conformant to bootstrap into a POSIX shell (or your preferred shell), even on Solaris and AIX. And if you're willing to stick to simple scripts or rigorously test across systems, sufficient for most tasks. Outside Linux-based systems it's usually ksh88, ksh93, pdksh, or some derivative. OTOH, for those who are only familiar with bash that may not be particularly helpful.

I've had more trouble, including bugs, with other utilities, like sed, tr, paste, etc. For shell portability it's usually niche stuff like "$@" expansion with empty lists, for example how it interacts with nounset or IFS, independent of POSIX mode.

Re: A surprise with how '#!' handles its program argument in practice

#102

Post by jborean93 »

SAI_Peregrinus wrote: post #89
rmunn wrote: post #31

Hit the link expecting to read about UTF-8 Byte Order Marks at the top of the file, so that the first few bytes aren't actually #! but 0xEF 0xBB 0xBF #! instead. Ran into this one just a few months ago when a coworker who uses Windows had checked a Bash script into the Git repo. His editor was configured to save files as "UTF-8 with BOM" and so we were getting errors that looked like "./doit.sh: line 1: #!/bin/bash:…

UTF-8 doesn't have a BOM. UTF-16 does. UTF-32 does. "UTF-8 with BOM" is not a standards-compliant text format, it's a proprietary binary format that happens to have a bunch of embedded UTF-8. Just because you can run `strings` on a file & get a bunch of text out doesn't mean it's a text file!

This seems a bit pedantic, while you may be correct (I honestly don't know what standard this is referring to) the UTF-8 BOM is a thing that some tools do know about. Even then in the context of OP's question the BOM with UTF-8 isn't the specific problem but rather how the shebang interpreter reads the actual ASCII byte sequences so a UTF-16 with a BOM "text" file would also fail.

Re: A surprise with how '#!' handles its program argument in practice

#103

Post by mbreese »

inetknght wrote: post #98
mbreese wrote: post #66

I do this all the time for python virtualenvs. Instead of calling ‘#!/usr/bin/env python3’ I will call ‘#!venv/bin/python3’. This way I don’t have to worry about whether the environment was activated or not.

This is terrible. I hope you're not adding a whole venv to version control. What if the user doesn't have a venv created? What if they created it in a different directory? What if they created a second venv and want to use that instead? What if the user uses `.venv` instead of `venv`? `#!/usr/bin/env python3` solves most of that.

No!

These are programs that are largely meant to have a run.sh or install.sh script run before the main script. If the venv doesn’t exist, the it is created there and requirements installed.

The main point is that I’m trading away some flexibility to keep my ENV clean. When I submit jobs on HPC clusters, keeping my environment clean tends to make things easier to troubleshoot.

If I’m switching between different programs or commonly piping data between two different programs with their own venvs, it can be easier to just run the associated python binary directly, rather than have to manage different venv environment activations.

Re: A surprise with how '#!' handles its program argument in practice

#104

Post by Ferret7446 »

Crestwave wrote: post #6
tombert wrote: post #2

Huh. I wish I had known this before. NixOS is annoying because everything is weird and symlinked and so I find myself fairly frequently making the mistake of writing `#!/bin/bash`, only to be told it can't find it, and I have to replace the path with `/run/current-system/sw/bin/bash`. Or at least I thought I did; apparently I can just have done `#!bash`. I just tested this, and it worked fine. You learn something new…

`#!/usr/bin/env bash` is the most portable form for executing it from $PATH

I raise you a https://www.felesatra.moe/blog/2021/07/03/portable-bash-sheb...

Pedantic, but "#!" and "portable" don't belong in the same sentence

Re: A surprise with how '#!' handles its program argument in practice

#105

Post by inetknght »

mbreese wrote: post #103
inetknght wrote: post #98

Earlier quoted context omitted.

This is terrible. I hope you're not adding a whole venv to version control. What if the user doesn't have a venv created? What if they created it in a different directory? What if they created a second venv and want to use that instead? What if the user uses `.venv` instead of `venv`? `#!/usr/bin/env python3` solves most of that.

No! These are programs that are largely meant to have a run.sh or install.sh script run before the main script. If the venv doesn’t exist, the it is created there and requirements installed. The main point is that I’m trading away some flexibility to keep my ENV clean. When I submit jobs on HPC clusters, keeping my environment clean tends to make things easier to troubleshoot. If I’m switching between different progr…

You can have your cake (use multiple venvs) and eat it (flexibility) too.

`source venv/bin/activate` from your .sh files will cause `/usr/bin/env python3` to use the python3 located in your venv. Switching between venvs is easy too. just call `deactivate` when one venv is activated. It drops you out of the venv. You can then cleanly `source venv2/bin/activate`.

Re: A surprise with how '#!' handles its program argument in practice

#106

Post by mbreese »

inetknght wrote: post #105
mbreese wrote: post #103

Earlier quoted context omitted.

No! These are programs that are largely meant to have a run.sh or install.sh script run before the main script. If the venv doesn’t exist, the it is created there and requirements installed. The main point is that I’m trading away some flexibility to keep my ENV clean. When I submit jobs on HPC clusters, keeping my environment clean tends to make things easier to troubleshoot. If I’m switching between different progr…

You can have your cake (use multiple venvs) and eat it (flexibility) too. `source venv/bin/activate` from your .sh files will cause `/usr/bin/env python3` to use the python3 located in your venv. Switching between venvs is easy too. just call `deactivate` when one venv is activated. It drops you out of the venv. You can then cleanly `source venv2/bin/activate`.

It feels like you're telling me that I'm holding my phone wrong.

I don't like sourcing things into my environment. I've worked this way for years. I think the idea of 'activating' and 'deactivating' an environment is an anti-pattern. But I also work on HPC clusters where all of the configuration about paths is handled by the environment. Because of this, I've learned the hard way that for my workflows, it's far too easy to have the wrong environment loaded with venvs and modules that it's often better to keep things explicitly defined. I don't like magic, so I explicitly state which venv my code (or occasionally other people's code) is loading from.

I sometimes will have to run multiple programs (that have their own venvs) and pipe data between them. If I have to source and deactivate different venvs for each tool, it just doesn't work right.

I think that's part of the power of virtualenv as a tool -- it's flexible in how it works. I can still use my explicit workflows with the same tooling as everyone else, and you can source your environments and keep happily coding along. For me, that's why I keep using them...

Re: A surprise with how '#!' handles its program argument in practice

#107

Post by alexey-salmin »

userbinator wrote: post #42

I wonder what the reason was for having the kernel handle this, instead of the shell? To allow programs besides the shell to execute interpreted scripts as if they were actual binaries? This is of course in stark contrast to dynamic linking, which is performed by a userspace program instead of the kernel, and much like the #!, this "interpreter"'s path is also hardcoded in dynamically linked binaries.

> This is of course in stark contrast to dynamic linking, which is performed by a userspace program instead of the kernel

I'm not sure what the contrast is. In both cases the interpreter lives in userspace, in both cases kernel finds it via the path hardcoded in the file: shebang for text, .interp for ELF

Re: A surprise with how '#!' handles its program argument in practice

#108

Post by jibal »

jmclnx wrote: post #60
jibal wrote: post #26

There's no security issue here. Certainly the OP hasn't explained why there is one.

There could be. Say you have '#!sh', someone could have put a broken 'sh' somewhere, that could be the first one the '#!' sees.

No, this is about relative paths, not searching $PATH ..., which it doesn't do. (That happens when people do '#!/usr/bin/env sh' ... the security issues with that are a separate discussion.)

Re: A surprise with how '#!' handles its program argument in practice

#109

Post by tracker1 »

Kind of an adjacent aside, I kind of love how many languages now work with shebangs for creating user scripts... Go, Rust, C#, Deno (TS/JS) etc.

The vast majority of my shell/user scripts are now in TS with Deno... The only thing I do wish worked better is that VS Code would look at the shebang to determine a file's language without a file extension.

Also works nicely with project automation and CI/CD integration as well... You can do a lot more, relatively easier than having to have knowledge of another language/tool you aren't as comfortable with.

Re: A surprise with how '#!' handles its program argument in practice

#110

Post by extraduder_ire »

1718627440 wrote: post #51

> You're using a suspiciously old browser Does the author not like Firefox or what?

That page[0] should explain the reasoning behind the redirect. There's a separate redirect[1] if you have a never version reported but are missing headers.

Seems to be a fairly custom attempt at keeping poorly written scrapers off the site. curl works just fine for me. (I found these redirect URLs by manually setting the user agent header in curl) Fun idea.

What's your browser's user agent? You might have an extension that alters yours to not report your browser's actual version number.

0: https://utcc.utoronto.ca/~cks/cspace-old-browser.html

1: https://utcc.utoronto.ca/~cks/cspace-no-sec-fetch.html

Post reply on HN