Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Four features that justify a new Unix shell

oilshell.org

Post reply on HN
Print view View original

91–100 of 185 posts

Re: Four features that justify a new Unix shell

#91

Post by chubot »

kbenson wrote: post #64
pierrebai wrote: post #60

I'm sorry to say to for me, shell scripting has lost. All my scripts are written in Python, which a thin shell script wrapper to launch it. Look at Oil. It tries hard to be bash compatible with optional buy-in to extension. Just use a superior language and give up on shell scripts, that's my advice.

I agree. Generally I find the approach to have a few tools that work well in their specific niches to be superior to having one tool for all jobs. I use a shell for shell stuff, I use a scripting language for most other stuff, and I might use a systems level compiled language (or a scripting language that calls into a compiled library) for more performance specific needs. If you're already within a specific area and…

the approach to have a few tools that work well in their specific niches to be superior to having one tool for all jobs.

Exactly, that is the point of shell.

Shell scripts are meant to invoke Python or Ruby programs.

sibling comment: https://news.ycombinator.com/item?id=24875932

earlier comment: https://news.ycombinator.com/item?id=24083764

Re: Four features that justify a new Unix shell

#92

Post by esaym »

FreshFries wrote: post #71
R0b0t1 wrote: post #66

Earlier quoted context omitted.

There is a better option and its name is Perl. Perl mixes the ins of shell scripts like easy argument and output passing with complex and easy to use control structures. You could use Python, but having been forced to use Perl extensively Perl is the superior choice for a complex shellscript-like workload. There's less boilerplate in Perl. There's a couple Python projects that come close like Fabric, but Perl was lit…

There is nothing in the (IT) world I fear more then... Perl projects. The shear madness it will leash upon you when the dependencies fail is maddening. Never will I voluntarily touch anything written in Perl ever again in my live. I've seen the power of Perl, the beauty and elegance of it, but everything was ruined by the absolute shit show when it comes to its packet managers. /rant.

I think the OP is mostly talking about "perl one liners" on the shell like this: https://catonmat.net/ftp/perl1line.txt

It is not that hard either (unless perl is completely foreign to you).

Ironically just today I wrote a python script (using concurrent.futures) to batch re-encode 10gb of mp3 podcasts into 3gb of opus files to save space. Before I deleted the old mp3's, I did a quick file count of mp3 and opus files only to have 5 extra opus files... I beat my head really good trying to glue a bunch of bash stuff together but I could not find where the 5 extra opus files were coming from.

I defected back to perl. All I needed was an alphabetical list of all the files (there were many sub-directories), perfect for a little perl oneliner:

  find /podcasts/ -iname "* \.ogg" | perl -nle 'if( $_ =~ /\/([\w-]+.\w\w\w)$/) { print "$1"}' | sort >ogg.txt
I did the same thing for * .mp3 files and then diffed the two files. I quickly noticed the extra 5 opus files were from me doing test encodes a couple of weeks ago that I forgot about.

Sure it looks hideous, but the explanation is easy: for 'perl -nle', the 'l' means process each line from stdin individually, the 'n' means no auto printing of the line, and the 'e' is basically the perl 'one liner' mode. Either way, each line comes in and is stored in '$_', I use a capturing regex to grab the file name without parent directory names, basically everything after the last '/' in a path that ends with a three letter extension (\w\w\w). The "captured" part of the regex ends up in '$1'. Also see 'perldoc run' for all the runtime one liner options.

Re: Four features that justify a new Unix shell

#93

Post by msla »

pierrebai wrote: post #60

I'm sorry to say to for me, shell scripting has lost. All my scripts are written in Python, which a thin shell script wrapper to launch it. Look at Oil. It tries hard to be bash compatible with optional buy-in to extension. Just use a superior language and give up on shell scripts, that's my advice.

Python has a REPL but it isn't sufficient to be a shell. Therefore, I can't write a program in the Python REPL just as I'm doing stuff normally and copy it into a file for further editing and polishing. I have to port whatever I did in the shell to whatever other language I'm using, which is more porting than I want to do for most shell scripts, especially ones which are mostly pipelines.

Heck, Python isn't even a good language for one-liners. Not saying it should be, but it further demonstrates that I can't use Python live the way I can use zsh live.

Re: Four features that justify a new Unix shell

#94

Post by dmix »

jonahx wrote: post #67
pierrebai wrote: post #60

I'm sorry to say to for me, shell scripting has lost. All my scripts are written in Python, which a thin shell script wrapper to launch it. Look at Oil. It tries hard to be bash compatible with optional buy-in to extension. Just use a superior language and give up on shell scripts, that's my advice.

The author addresses this argument: > However, Python and Ruby aren't good shell replacements in general. Shell is a domain-specific language for dealing with concurrent processes and the file system. But Python and Ruby have too much abstraction over these concepts, sometimes in the name of portability (e.g. to Windows). They hide what's really going on. From: http://www.oilshell.org/blog/2018/01/28.html#i-dont-unde…

Sounds like we need a new standard!

Re: Four features that justify a new Unix shell

#95

Post by chubot »

R0b0t1 wrote: post #66
pierrebai wrote: post #60

I'm sorry to say to for me, shell scripting has lost. All my scripts are written in Python, which a thin shell script wrapper to launch it. Look at Oil. It tries hard to be bash compatible with optional buy-in to extension. Just use a superior language and give up on shell scripts, that's my advice.

There is a better option and its name is Perl. Perl mixes the ins of shell scripts like easy argument and output passing with complex and easy to use control structures. You could use Python, but having been forced to use Perl extensively Perl is the superior choice for a complex shellscript-like workload. There's less boilerplate in Perl. There's a couple Python projects that come close like Fabric, but Perl was lit…

I think Perl needs a real pipeline syntax, among other things: http://www.oilshell.org/blog/2018/01/28.html#are-you-reinven...

That said, Oil is somewhat influenced by Perl: https://www.oilshell.org/release/0.8.3/doc/language-influenc...

Also note this part of the original blog post, Python/JS/Ruby vs Perl/PHP: http://www.oilshell.org/blog/2020/10/osh-features.html#oil-l...

I recently bought a copy of the Camel book and noticed the exact same thing that Steve Yegge pointed out here:

Perl's references are basically pointers. As in, C-style pointers. You know. Addresses. Machine addresses. What in the flip-flop are machine addresses doing in a "very high-level language (VHLL)", might you ask? Well, gosh, what they're doing is taking up about 30% of the space in all Perl documentation worldwide.

https://sites.google.com/site/steveyegge2/ancient-languages-...

This is really a huge, unnecessary wart in the language.

The other wart I pointed out in the post is that Perl 5 does dynamic parsing of its own code (parsing that depends on the value of variables), despite the book complaining about the same issue in shell. In contrast, Raku and Oil are statically parsed.

----

I have a lot of respect for Perl, but there's a reason that Raku exists (and again from the FAQ: Raku and Python 3 are both worse shell-like languages than their predecessors.)

Re: Four features that justify a new Unix shell

#96

Post by esaym »

johnisgood wrote: post #78
FreshFries wrote: post #71

Earlier quoted context omitted.

There is nothing in the (IT) world I fear more then... Perl projects. The shear madness it will leash upon you when the dependencies fail is maddening. Never will I voluntarily touch anything written in Perl ever again in my live. I've seen the power of Perl, the beauty and elegance of it, but everything was ruined by the absolute shit show when it comes to its packet managers. /rant.

I hate everything there is to hate about CPAN. I also hate how Perl libraries are handled. I have to have like 5 lines of code related to it in ~/.bashrc or ~/.bash_profile . Bleh. Perl libraries are always a problem when I want my program to run on other machines. No thank you.

I find this to be a weird statement. I've never really fought with "cpan" but have had huge fights with node and python packages.

The old school "cpan" command has not been recommended to be used for nearly a decade now. Maybe that is your issue?

Everyone just uses cpan-minus: http://cpanmin.us/ or "cpanm". It will install anything, if you have write access to the installed perl location, it will install globally, otherwise it installs packages into "~/perl5" then just do something like "export PERL5LIB=~/perl5/lib/perl5/" in your bashrc.

Or maybe you are talking about having to compile code in certain packages? Alot of stuff using ssl has C code which links against openssl which is always a pain.

Not sure what other issues you might be alluding to... There are some "packaging" tools to try to bundle and pin libraries, kind of like venv for python. I've never messed with them though. We just compile the latest perl binary and install all our needed cpan modules globally. Never worry about pinning since nothing is ever updated on cpan anymore...

Re: Four features that justify a new Unix shell

#97

Post by CameronNemo »

chubot wrote: post #90
pierrebai wrote: post #60

I'm sorry to say to for me, shell scripting has lost. All my scripts are written in Python, which a thin shell script wrapper to launch it. Look at Oil. It tries hard to be bash compatible with optional buy-in to extension. Just use a superior language and give up on shell scripts, that's my advice.

(author here) I do use Python. In fact I wrote something like 30K lines of Python for Oil. The question isn't shell OR Python. My shell scripts call Python scripts, many of white I wrote myself. That is working as intended. They also call C programs, C++ programs and R programs. And put JavaScript programs in various places. ---- I guess this is a huge misconception about shell that I have to write a blog post about.…

>When you program in shell, gcc, git, pip, npm, markdown, rsync, diff, perf, strace, etc. are part of your "standard library".

I don't think this is a good characterization at all... Shell does not have git functionality (for example) built in. That is a dependency just like it would be for a python or rust project.

Re: Four features that justify a new Unix shell

#98

Post by IggleSniggle »

Doesn't TCL (and tclsh) handle these top four features quite elegantly? I understand why people shudder at TCL, but they are mixing up the elegance of the language with the dysfunction of the situation it was pulled in to address.

Yes, when you start gluing random stuff together it gets ugly. This is true in real-life too. That's the situation.

TCL, from the "commands and strings" handles this quite elegantly with a kind of Haskell-like power. And you can just use it as your shell, because it is designed to run commands! And if you are on macOS or Linux, there's a good chance you even already have it installed. So go give it a try.

I think perl comes at this from "the other end," making a general-use language into a shell-convenient package, but I don't have enough experience on it to comment. Nonetheless, I suspect the horrors of perl are somewhat similar in nature: Too many blackbox systems that must be glued together.

Re: Four features that justify a new Unix shell

#99

Post by l0b0 »

chubot wrote: post #56
l0b0 wrote: post #19

There are some good ideas in here, but `--qsn` is described[1] like this: > Print filenames ONE PER LINE. If a name contains a newline or other special char, it's QSN-encoded like 'multi-line \n name with NUL \0 byte' This is a bad idea. There is a solution which already works. Just terminate every filename with NUL and you're done. Trying to make something "sorta kinda human readable" is a mistake when dealing with…

The paragraph right below that mentions that Oil has "read -0", which consumes the find -print0 input. I also link to my Git Log in HTML post [1] from 3 years ago, which is ENTIRELY about the NUL byte solution :) ----- Shell scripts can use both formats, but the advantage to QSN is that it preserves the line-based nature of shell. Say I want to use wc -l, awk, or grep. Then the QSN-lines format is better than the NUL…

> The paragraph right below that mentions that Oil has "read -0", which consumes the find -print0 input.

Yeah, I read that ("read -0", for the record, is an excellent idea). QSN for filenames still a bad idea. Your wc, awk, grep etc commands will now have to decode the stream after splitting it. Taking it to the extreme, it's like mixing JSON and XML (or CSV and TSV) because some things work better in one or the other.

Right now, even in Bash, I can use newline-terminated tokens and accept that newlines are going to screw up everything (which is fine if I control the filenames), or I can use NUL-terminated tokens and force my code to handle absolutely all possible inputs. No extra parsing library necessary.

As long as filenames can contain any character except slash and NUL, terminating them by NUL is a simple solution (at least when using GNU tools).

> Shell scripts can use both formats, but the advantage to QSN is that it preserves the line-based nature of shell.

Except when your tokens can contain newlines, which we're stuck with for the foreseeable future.

> Say I want to use wc -l, awk, or grep. Then the QSN-lines format is better than the NUL format.

Can't agree, for the reasons above.

> Also, you can transmit a series of say SHA256 checksums in binary format with QSN-lines, or some other binary format.

> Even entire JPG files, audio clips, wasm files, whatever. It's "8-bit clean" (and so are Oil strings).

That's a completely different problem set. You could use existing simple encodings like base64. "Why use QSN over base64" would be an interesting blog post.

In any case, I admire the courage to try to improve the state of the art! I would absolutely love to see something better than POSIX/Bash as the baseline. At this point I suspect we'll need to ignore those and go for something radically different like PowerShell to regain sanity.

Re: Four features that justify a new Unix shell

#100

Post by 7thaccount »

MaikuMori wrote: post #75
7thaccount wrote: post #30

Earlier quoted context omitted.

It's interesting until you run up against speed limitations both left and right. I still use it for some work and enjoy it, but it is by far the slowest technology I've used.

Believe it or not you have to disable the progress bar to speed some things up. Downloads alone are at least 10x faster if you disable progress bar.

File read and writes are also glacially slow by all the common cmdlets. Even if you use .NET in your code and make it far more verbose it CRAWLS.

Someone on here previously showed a trick to get some speedups, but it is still slow.

Post reply on HN