Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pnut: A C to POSIX shell compiler you can trust

pnut.sh

Post reply on HN
Print view View original

91–100 of 124 posts

Re: Pnut: A C to POSIX shell compiler you can trust

#91

Post by chasil »

AdmiralAsshat wrote: post #85
laurenth wrote: post #78

Earlier quoted context omitted.

From our experience, ksh is generally faster, and dash sits between ksh and bash. One reason is that dash stores variables using a very small hash table with only 37 entries[0] meaning variable access quickly becomes linear as memory usage grows. But even with that, dash is still surprisingly fast -- when compiling `pnut.c` with `pnut.sh`, dash comes in second place: ksh93: 31s dash: 1m06s bash: 1m19s zsh: >15m [0]:…

People still use KornShell?

All of Android is still based on a pdksh-derivative known as mksh, which is an enormous install base.

http://www.mirbsd.org/mksh.htm

OpenBSD switched their default shell to their own pdksh-derivative known as oksh.

https://github.com/ibara/oksh

There was an effort to (re)start ksh93 development, but AT&T halted this effort. The bugfixes from the failed effort have moved back into Korn's last release.

https://github.com/ksh93/ksh/tree/dev

Re: Pnut: A C to POSIX shell compiler you can trust

#92

Post by Q-Q3 »

voidUpdate wrote: post #39

When I'm told that "I can trust" something that I feel like I had no reason to distrust, it makes me feel even more suspicious of it

Hi there! I believe the mention of "trust" is related to the paper Reflections on Trusting Trust by Ken Thompson https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref... Though I do think the tagline used could definitely be improved from a marketing standpoint.

Re: Pnut: A C to POSIX shell compiler you can trust

#93

Post by chasil »

throwaway2037 wrote: post #75
chasil wrote: post #72

Earlier quoted context omitted.

It would actually be interesting to see how much faster dash is than everything else.

Why is Dash frequently touted as so much faster than Bash? What is different?

On rhel9, this is a list of my installed shells. You might notice that dash is smaller than ls (and the rest of the shells).

  $ ll /bin/bash /bin/dash /bin/ksh93 /bin/ls /bin/mksh
  -rwxr-xr-x. 1 root root 1389064 May  1 00:59 /bin/bash
  -rwxr-xr-x. 1 root root  128608 May  9  2023 /bin/dash
  -rwxr-xr-x. 1 root root 1414912 Apr  9 07:26 /bin/ksh93
  -rwxr-xr-x. 1 root root  140920 Apr  8 08:20 /bin/ls
  -rwxr-xr-x. 1 root root  325208 Jan  9  2022 /bin/mksh

  $ rpm -qi dash | tail -4
  Description :
  DASH is a POSIX-compliant implementation of /bin/sh that aims to be as small as
  possible. It does this without sacrificing speed where possible. In fact, it is
  significantly faster than bash (the GNU Bourne-Again SHell) for most tasks.

Re: Pnut: A C to POSIX shell compiler you can trust

#94

Post by cb321 »

laurenth wrote: post #78
chasil wrote: post #72

Earlier quoted context omitted.

It would actually be interesting to see how much faster dash is than everything else.

From our experience, ksh is generally faster, and dash sits between ksh and bash. One reason is that dash stores variables using a very small hash table with only 37 entries[0] meaning variable access quickly becomes linear as memory usage grows. But even with that, dash is still surprisingly fast -- when compiling `pnut.c` with `pnut.sh`, dash comes in second place: ksh93: 31s dash: 1m06s bash: 1m19s zsh: >15m [0]:…

For me `dash` compiles in just a few seconds. If you link to a 1-line problem (here, #define VTABSIZE 39), then why not boost that to 79 or 113, say, re-compile the shell and re-run your benchmark? Might lead to a change in upstream that could benefit everyone.

Re: Pnut: A C to POSIX shell compiler you can trust

#96

Post by chasil »

cb321 wrote: post #94
laurenth wrote: post #78

Earlier quoted context omitted.

From our experience, ksh is generally faster, and dash sits between ksh and bash. One reason is that dash stores variables using a very small hash table with only 37 entries[0] meaning variable access quickly becomes linear as memory usage grows. But even with that, dash is still surprisingly fast -- when compiling `pnut.c` with `pnut.sh`, dash comes in second place: ksh93: 31s dash: 1m06s bash: 1m19s zsh: >15m [0]:…

For me `dash` compiles in just a few seconds. If you link to a 1-line problem (here, #define VTABSIZE 39), then why not boost that to 79 or 113, say, re-compile the shell and re-run your benchmark? Might lead to a change in upstream that could benefit everyone.

Or rework the array so realloc() can expand its size?

Re: Pnut: A C to POSIX shell compiler you can trust

#97

Post by cb321 »

chasil wrote: post #96
cb321 wrote: post #94

Earlier quoted context omitted.

For me `dash` compiles in just a few seconds. If you link to a 1-line problem (here, #define VTABSIZE 39), then why not boost that to 79 or 113, say, re-compile the shell and re-run your benchmark? Might lead to a change in upstream that could benefit everyone.

Or rework the array so realloc() can expand its size?

Yes.. Another fine idea, just more work than a 2 character edit. :-)

Re: Pnut: A C to POSIX shell compiler you can trust

#98

Post by laurenth »

iod wrote: post #87

I am sorry if this comes off to be negative, but with every example provided on the site, when compiled and then fed into ShellCheck¹, generates warnings about non-portable and ambiguous problems with the script. What exactly are we supposed to trust? ¹ https://www.shellcheck.net

It seems ShellCheck errs on the side of caution when checking arithmetic expansions and some of its recommendations are not relevant in the context they are given. For example, on `cat.sh`, one of the lines that are marked in red is:

  In examples/compiled/cat.sh line 7:
    : $((_$__ALLOC = $2)) # Track object size
      ^-- SC1102 (error): Shells disambiguate $(( differently or not at all. For $(command substitution), add space after $( . For $((arithmetics)), fix parsing errors.
      ^-----------------^ SC2046 (warning): Quote this to prevent word splitting.
        ^--------------^ SC2205 (warning): (..) is a subshell. Did you mean [ .. ], a test expression?
                   ^-- SC2283 (error): Remove spaces around = to assign (or use [ ] to compare, or quote '=' if literal).
                     ^-- SC2086 (info): Double quote to prevent globbing and word splitting.
It seems to be parsing the arithmetic expansion as a command substitution, which then causes the analyzer to produce errors that aren't relevant. ShellCheck's own documentation[0] mention this in the exceptions section, and the code is generated such that quoting and word splitting are not an issue (because variables never contain whitespace or special characters).

It also warns about `let` being undefined in POSIX shell, but `let` is defined in the shell script so it's a false positive that's caused by the use of the `let` keyword specifically.

If you think there are other issues or ways to improve Pnut's compatibility with Shellcheck, please let us know!

0: https://www.shellcheck.net/wiki/SC1102

Re: Pnut: A C to POSIX shell compiler you can trust

#99

Post by laurenth »

taviso wrote: post #79
laurenth wrote: post #3

Earlier quoted context omitted.

Author here, Because all shell variables in code generated by pnut are numbers, variables never contain whitespace or special characters and don't need to be quoted. We considered quoting all variable expansions as this is generally seen as best practice in shell programming, but thought it hurt readability and decided not to. If you think there are other issues, please let me know!

I think they're talking about the cp example, doesn't seem like it would handle filenames with spaces! Super neat project, btw!

You're right, thanks for the bug report. It should now be fixed :)

Re: Pnut: A C to POSIX shell compiler you can trust

#100

Post by laurenth »

MobiusHorizons wrote: post #76
luism6n wrote: post #12

Earlier quoted context omitted.

I'm not the OP, but I think the goal is to make it cross architecture. Cross platform C compiler would give you cross OS compatibility, but chip architecture would still be fixed, I think. I.e., you can take your compiled.sh and run in an obscure processor with an obscure OS, as long as it's POSIX, it should work...

I believe the goal is to defeat the compiler trust thought exercise where a malicious compiler could replicate itself when being asked to compile the compiler. Since this produces human readable code instead of assembly, the idea is it allows bootstrapping a trusted compiler, since pnut.sh and any output shell executables are directly auditable. I suppose the trust moves to the shell executable then, but at least you…

That's the idea!

As you point out, it moves the trust from the binary to the shell executable, but the shell is already a key piece of any build process and requires a minimum level of trust. The technique of bootstrapping on multiple shells and comparing the outputs is known as Double Diverse Compiling[0] and we think POSIX shell is particularly suited for this use case since it has so many implementations from different and likely independent sources.

The age and stability of the POSIX shell standard also play in our favor. Old shell binaries should be able bootstrap Pnut, and those binaries may be less likely to be compromised as the trusting trust attack was less known at that time, akin to low-background steel[1] that was made before nuclear bombs contaminated the atmosphere and steel produced after that time.

0: https://dwheeler.com/trusting-trust/ 1: https://en.wikipedia.org/wiki/Low-background_steel

Post reply on HN