Four features that justify a new Unix shell • Hacker News Board

Re: Four features that justify a new Unix shell

#161

Post by indigo945 » Sat, Oct 24, 2020, 1:02 PM UTC

IggleSniggle wrote: post #160

qznc wrote: post #143

Earlier quoted context omitted.

TCL has a very simple model of a shell "REPL": interpret string as expression and evaluate. In contrast, a Unix shell has job control, stderr, pipes, etc.

I’m struggling to understand how you don’t get these features with tcl / tclsh. You get job control, familiar access to your file system and the usual conventions around command execution of binaries in your PATH (like ps, kill, etc), you have stderr, you have pipes... ...but you also have a language that is immediately polyglot because of the way it uses strings. Each string reference is analogous to a file-descript…

But you can't connect programs together via pipes from within TCL, except by doing a terrible hand-rolled implementation that does half the things that bash gives you for free. And arguably, the ability to connect programs together like that is the most powerful feature of any modern shell.

Re: Four features that justify a new Unix shell

#162

Post by acdha » Sat, Oct 24, 2020, 1:35 PM UTC

cle wrote: post #123

acdha wrote: post #116

Earlier quoted context omitted.

That’s not a very convincing post because it’s a bit too shallow: > I encountered a nice blog post, Replacing Shell Scripts with Python, which, in my opinion, inadvertently proves the opposite point. The Python version is longer and has more dependencies. In other words, it's more difficult to write and maintain. The Python version uses only the standard library, so it doesn’t make sense to worry about dependencies i…

This. The Bash script actually has more dependencies, it relies on a number of external programs (ps, kill, mkdir, sort, ls, cp, echo). What versions are on your system? What versions on the systems of the people running the script? Do they support the same features? If you're running on a Mac are you using the Mac-shipped programs or the GNU coreutils from homebrew? Also, that Bash script that the author is defendin…

Good catch: I’ve written many thousands of lines of shell scripts over the years and that’s exactly the kind of rake in the grass which is too easy to miss. Using set -eu -o PIPEFAIL helps as does shellcheck but it’s still easier than it should be to have unexpected behavior from something you thought was well tested.

About 15 years ago I switched to Python for anything which doesn’t fit on a single screen or uses any advanced feature and have had zero reasons to regret that decision. It’s especially good for anything you use with other people since you spend your time talking about features rather than how to trick the shell into working correctly.

Re: Four features that justify a new Unix shell

#163

Post by IggleSniggle » Sat, Oct 24, 2020, 2:08 PM UTC

indigo945 wrote: post #161

IggleSniggle wrote: post #160

Earlier quoted context omitted.

I’m struggling to understand how you don’t get these features with tcl / tclsh. You get job control, familiar access to your file system and the usual conventions around command execution of binaries in your PATH (like ps, kill, etc), you have stderr, you have pipes... ...but you also have a language that is immediately polyglot because of the way it uses strings. Each string reference is analogous to a file-descript…

But you can't connect programs together via pipes from within TCL, except by doing a terrible hand-rolled implementation that does half the things that bash gives you for free. And arguably, the ability to connect programs together like that is the most powerful feature of any modern shell.

`open |myProgram` returns a file-descriptor for a pipe (that you can explicitly open for read, write, or read/write), and `exec ls |grep {myPattern}` will pipe the output of ls into grep...okay nevermind I do see your point.

    p1 | p2 | p3 > file

is certainly nicer for interactive use, although I would argue that a hand-rolled pipe operator to give you the same in tcl comes with some pleasant fp benefits, and it’s only a few lines to define it to get the above, or supply alternative syntax like

| {p1} {p2} {p3} {[open file w]}

or if you want to also define the redirect,

| {p1} {p2} {p3} {> file}

where each portion of the pipe can be much more clearly manipulated in-stream, and with whatever interpreter, before returning the pipeable.

This gets especially nice for gluing when you want to run scripts in multiple interpreters but they are short and you want to maintain the context for readability between multiple execution contexts:

    | {node -e {
        console.log(“start”)
      }}
      {DSL1 {
        MyDSLScriptHere
      }}
      {grep {whatever}}
      {> file}

Sometimes the extra context hurts readability, but sometimes it’s invaluable.

Re: Four features that justify a new Unix shell

#164

Post by johnisgood » Sat, Oct 24, 2020, 2:37 PM UTC

esaym wrote: post #96

johnisgood wrote: post #78

Earlier quoted context omitted.

I hate everything there is to hate about CPAN. I also hate how Perl libraries are handled. I have to have like 5 lines of code related to it in ~/.bashrc or ~/.bash_profile . Bleh. Perl libraries are always a problem when I want my program to run on other machines. No thank you.

I find this to be a weird statement. I've never really fought with "cpan" but have had huge fights with node and python packages. The old school "cpan" command has not been recommended to be used for nearly a decade now. Maybe that is your issue? Everyone just uses cpan-minus: http://cpanmin.us/ or "cpanm". It will install anything, if you have write access to the installed perl location, it will install globally, ot…

> The old school "cpan" command has not been recommended to be used for nearly a decade now. Maybe that is your issue?

I did not know that it was not supposed to be used, so that might explain my issues, but at the same time it did work for me on my machine, but had a hard time running my Perl script on a VPS, because I had to set everything up again and for some reason I ran into some errors.

I often ran into packages that failed to compile, too, yeah.

> Never worry about pinning since nothing is ever updated on cpan anymore...

Nothing makes it kind of sad. :/

Re: Four features that justify a new Unix shell

#165

Post by vips7L » Sat, Oct 24, 2020, 2:38 PM UTC

chubot wrote: post #127

oblio wrote: post #47

Earlier quoted context omitted.

On the other hand, for interoperability, passing around behavior is horrible. It's either a security risk or a compatibility (forward/backward) risk or both. We should try to pass around just state and keep objects strictly for code.

Right, I'm not really familiar with PowerShell, but my understanding is that the objects in object pipelines are literally .NET objects with methods on them. So the entire shell script is confined to the .NET VM? In that case, I would hesitate to even call it a shell in the traditional sense. Shell has a kind of code data code data architecture, i.e. programs in different languages processing standard language-indepe…

I think you need to go actually spend some time with powershell before you make more assumptions about it and it's behavior.

> Shell has a kind of code data code data architecture, i.e. programs in different languages processing standard language-independent data formats (lines of text, JSON, HTML, QSN, etc.)

Powershell natively understands these with builtins. You can convert from and to json, xml, html, csv, and excel.

> What if I want to pass some data to R and plot it?

Just pipe it to your R program like you would any other shell.

Re: Four features that justify a new Unix shell

#166

Post by coldtea » Sat, Oct 24, 2020, 2:48 PM UTC

boogies wrote: post #73

coldtea wrote: post #61

Earlier quoted context omitted.

> Do one thing and do it well. That's how we got in today's mess. Do the things people want in a coherent, cohesive, well thought out way - don't combine N independent programs that can barely took to each other with stringly typing and ad-hoc parsing...

Sounds like you’d prefer Python.

Well, I do Python for decades :-)

But I would be OK with

(a) shell programs that talk to each other through pipes with something structured -- e.g. depending on the program with something like a pandas dataframe(binary) a parsable tree (e.g. json-like) or csv output and input -- as opposed to random text people try to parse with cut -f, regexes and so on.

(b) shell programs that share the same flags for the same things (not e.g. -o/--output for output in one program and -w/--write-file in another) and only specialize in the behavior they need...

and several other things besides...

Re: Four features that justify a new Unix shell

#167

Post by mikepurvis » Sat, Oct 24, 2020, 2:53 PM UTC

d0mine wrote: post #153

mikepurvis wrote: post #102

Earlier quoted context omitted.

Even something that should be as simple as piping output from one process to another in Python is a nightmare, and fraught with opportunities to shoot yourself in the foot with buffering deadlocks and other nonsense. The consequence of this is that many of my projects which require these overlapping domains (think stuff like build automation) ends up being the two interleaved— either an outer shell script that calls…

Here's how to pipe two commands in Python: import subprocess subprocess.check_call("command1 | command2", shell=True) Depending on your use case there are other options e.g., fabric http://fabfile.org plumbum Use shell for what is really good for: concise DSL for running commands (one-liners). Leave complex logic (branches, explicit loops) for sane general purpose languages such as Python.

Right, and I do that, but it's an obvious hack; it causes a separate sh process to spawn, you open yourself to shell injection issues if you're not careful, teeing the pipe is trickier than it should be, etc etc.

Re: Four features that justify a new Unix shell

#168

Post by onlyrealcuzzo » Sat, Oct 24, 2020, 3:24 PM UTC

edgyquant wrote: post #137

PurpleFoxy wrote: post #125

Earlier quoted context omitted.

IMO as soon as you feel the need to create a file for your shell commands, it’s time to use another language. Bash is great for quick jobs like running ffmpeg on every file in a folder or counting the lines in a file but it sucks as a programming language.

It sucks as a programming language because it’s a shell. No I’ve written rather large scripts that were robust but had no need to be written in a programming language. If your gluing programs together shell script are literally designed for that. If you’re doing lots of arithmetic and huge amounts of variable logic then, yes, you should switch to a programming language. Each tool has its place but I’ve seen people wi…

^ and the Python script likely has many more bugs.

One of the best things about gluing together battle tested Unix programs is that you'll only have bugs in your glue, not in the tricky logic that's more error prone.

Re: Four features that justify a new Unix shell

#169

Post by d0mine » Sat, Oct 24, 2020, 3:47 PM UTC

mikepurvis wrote: post #167

d0mine wrote: post #153

Earlier quoted context omitted.

Here's how to pipe two commands in Python: import subprocess subprocess.check_call("command1 | command2", shell=True) Depending on your use case there are other options e.g., fabric http://fabfile.org plumbum Use shell for what is really good for: concise DSL for running commands (one-liners). Leave complex logic (branches, explicit loops) for sane general purpose languages such as Python.

Right, and I do that, but it's an obvious hack; it causes a separate sh process to spawn, you open yourself to shell injection issues if you're not careful, teeing the pipe is trickier than it should be, etc etc.

you might have known this but the practice shows that It has to be repeated: there is no shell injection issues with literal string in your own code.

YMMV but for those rare cases when I care about the shell injection, I just don't use shell and run the command directly in Python. Combination of shell one-liners and the main logic in Python works well in practice.

Re: Four features that justify a new Unix shell

#170

Post by jhayward » Sat, Oct 24, 2020, 4:22 PM UTC

pmoriarty wrote: post #83

pierrebai wrote: post #60

I'm sorry to say to for me, shell scripting has lost. All my scripts are written in Python, which a thin shell script wrapper to launch it. Look at Oil. It tries hard to be bash compatible with optional buy-in to extension. Just use a superior language and give up on shell scripts, that's my advice.

I've actually started to transition my shell scripts to eLisp for better integration in to Emacs and eshell. As a Lisp, eLisp is not the greatest, but I'd still much rather use it than Python. I also don't want to sit and twiddle my thumbs while a Python script takes its sweet time in loading. Slow startup time is the kiss of death for most shell scripts.

You find Emacs to have a faster startup time than Python?