I still have high hopes for Lush, the Lisp Universal SHell[0].
Four features that justify a new Unix shell
121–130 of 185 posts
Re: Four features that justify a new Unix shell
#122Earlier quoted context omitted.
There is a better option and its name is Perl. Perl mixes the ins of shell scripts like easy argument and output passing with complex and easy to use control structures. You could use Python, but having been forced to use Perl extensively Perl is the superior choice for a complex shellscript-like workload. There's less boilerplate in Perl. There's a couple Python projects that come close like Fabric, but Perl was lit…
A lot of people have a knee-jerk negative reaction to the name perl without understanding that perl chewed up and spit out this particular problem space. You probably don't want to develop your next webapp with a team of perl coders, but the language has its niche.
I've switched most of my projects to golang these days, but even as recently as a couple of years ago I was writing pretty big sites with Perl.
Using https://metacpan.org/pod/CGI::Application to provide the framework, along with similar small frameworks - e.g. a perl version of sinatra.
Golang managed to persuade people to write tests, by making it easy and self-contained. I'd suggest with Test::More that Perl did something similar. Almost all the modules you'd find on CPAN are full of test-cases.
There might well be thinks we can argue about with Perl, but it is definitely true that it has/had one of the biggest and most consistent set of extensions out there. CPAN has held up pretty well, compared to some of the later alternatives (such as node modules, ruby gems, etc). One reason that I was able to work with Perl so easily was because I could find Stripe integrations, and similar with ease.
Re: Four features that justify a new Unix shell
#123Earlier quoted context omitted.
The author addresses this argument: > However, Python and Ruby aren't good shell replacements in general. Shell is a domain-specific language for dealing with concurrent processes and the file system. But Python and Ruby have too much abstraction over these concepts, sometimes in the name of portability (e.g. to Windows). They hide what's really going on. From: http://www.oilshell.org/blog/2018/01/28.html#i-dont-unde…
That’s not a very convincing post because it’s a bit too shallow: > I encountered a nice blog post, Replacing Shell Scripts with Python, which, in my opinion, inadvertently proves the opposite point. The Python version is longer and has more dependencies. In other words, it's more difficult to write and maintain. The Python version uses only the standard library, so it doesn’t make sense to worry about dependencies i…
This.
The Bash script actually has more dependencies, it relies on a number of external programs (ps, kill, mkdir, sort, ls, cp, echo). What versions are on your system? What versions on the systems of the people running the script? Do they support the same features? If you're running on a Mac are you using the Mac-shipped programs or the GNU coreutils from homebrew?
Also, that Bash script that the author is defending has a subtle bug in it, because Bash is subtle. The bug is here:
$APP $filename >"${output_dir}/summary_${name}.txt"
if [ $? != 0 ]; then
echo "Error $? in app"
fi
Which is exactly the kind of crap that everyone's talking about when saying that we should avoid Bash scripts. It's full of landmines like this.
Bash is great for interactive work on the terminal. It is not great for writing correct, maintainable programs.
Re: Four features that justify a new Unix shell
#124Earlier quoted context omitted.
I think the OP is mostly talking about "perl one liners" on the shell like this: https://catonmat.net/ftp/perl1line.txt It is not that hard either (unless perl is completely foreign to you). Ironically just today I wrote a python script (using concurrent.futures) to batch re-encode 10gb of mp3 podcasts into 3gb of opus files to save space. Before I deleted the old mp3's, I did a quick file count of mp3 and opus files…
If I've not wildly misunderstood what you're doing with that line (and apologies if I have), could you not just stick `-printf '%f\n'` on the end of the `find` command? (By definition, since you're looking for `.ogg` files, everything `find` finds will have a path that ends in a three letter extension.)
Ah, that is a good one! I've never even messed with the -prinf options.
Re: Four features that justify a new Unix shell
#125I am very sympathetic to this. I like the concept of a shell (there's a place for light scripting that doesn't involve python or perl or ruby), and appreciate what bash can do, but after encountering enough of the "gotchas" and just ugly behavior, I'm not really motivated to master it. The biggest reason to learn it seems to be the fact that everybody uses it and has done so for years. We need something akin to Jupyt…
The biggest reason to master the shell (I think) is that if you will discover a shocking amount of things you can quickly accomplish piping between the standard Unix binaries (essentially the standard library).
IMO as soon as you feel the need to create a file for your shell commands, it’s time to use another language. Bash is great for quick jobs like running ffmpeg on every file in a folder or counting the lines in a file but it sucks as a programming language.
Re: Four features that justify a new Unix shell
#126a shell doesn't have to be a programming language as well. I just need it to get around the fs and execute a program
Re: Four features that justify a new Unix shell
#127Earlier quoted context omitted.
An object should have identify, state and behaviour. JSON only encodes state.
On the other hand, for interoperability, passing around behavior is horrible. It's either a security risk or a compatibility (forward/backward) risk or both. We should try to pass around just state and keep objects strictly for code.
Right, I'm not really familiar with PowerShell, but my understanding is that the objects in object pipelines are literally .NET objects with methods on them.
So the entire shell script is confined to the .NET VM?
In that case, I would hesitate to even call it a shell in the traditional sense.
Shell has a kind of code data code data architecture, i.e. programs in different languages processing standard language-independent data formats (lines of text, JSON, HTML, QSN, etc.)
It's more like functional programming, where functions stand alone. (And note Oil is in a very OO style, because it deals with significant program state, so I'm not against objects. Right tool for the right job.)
-----
Other questions:
What if I want to pass some data to R and plot it? Or throw away some outliers with a little formula? I now have to figure out how to serialize those objects. Or do I have to write an R interpreter for the .NET VM? :)
What about splitting the pipeline over two different machines? I can do "ssh user@host find / -type f" trivially in shell. And I do this in practice, e.g. in Oil's continuous build: http://travis-ci.oilshell.org/jobs/
"Distributed objects" have proven to be a bad idea. "Real" (large scale, deployed) distributed systems are architected more like Unix than Windows.
The point of shell is to integrate disparate tools, so if there are some privileged tools in the .NET VM, and then some other tools that require a lot of work to get at, then that misses the point IMO.
Re: Four features that justify a new Unix shell
#128How about Powershell https://github.com/PowerShell/PowerShell I can't call myself a fan of powershell but if everyone switched I'd get used to it. sh and bash both seem like they should die in a fire. They are full of foot guns that end up costing millions in breaches and lost data. The space think even bit Apple back in the day, they had some OS upgrade script that ended up deleting your entire hard drive if there w…
sh and bash both seem like they should die in a fire. They are full of foot gun
Right, that is the point of Oil. The post describes 4 footguns that you can now avoid. Moreoever, you can run your existing shell scripts with Oil first, and gradually move away from the dangerous style.
Rewriting even 500 lines of shell in Python is not a pleasant task (even if you think Python is better for the task).
Re: Four features that justify a new Unix shell
#129Earlier quoted context omitted.
I think having both is a good idea. Sometimes maybe you get your list of path names from a Here-document, for example.
For sure we need both human- and machine-readable stream formats, but I don't really see how QSN has any advantage in that space over `printf '%q\n'` and NUL-terminated strings, respectively.
I actually considered that, but it doesn't work.
https://github.com/oilshell/oil/wiki/Shell-Almost-Has-a-JSON...
Single quoted strings in shell can't represent arbitrary strings either.
And the %q format is actually different from the format ${x@Q} emits. QSN is a well-specified format.
Re: Four features that justify a new Unix shell
#130Earlier quoted context omitted.
The paragraph right below that mentions that Oil has "read -0", which consumes the find -print0 input. I also link to my Git Log in HTML post [1] from 3 years ago, which is ENTIRELY about the NUL byte solution :) ----- Shell scripts can use both formats, but the advantage to QSN is that it preserves the line-based nature of shell. Say I want to use wc -l, awk, or grep. Then the QSN-lines format is better than the NUL…
> The paragraph right below that mentions that Oil has "read -0", which consumes the find -print0 input. Yeah, I read that ("read -0", for the record, is an excellent idea). QSN for filenames still a bad idea. Your wc, awk, grep etc commands will now have to decode the stream after splitting it. Taking it to the extreme, it's like mixing JSON and XML (or CSV and TSV) because some things work better in one or the othe…
A bunch of points:
(1) Except when your tokens can contain newlines, which we're stuck with for the foreseeable future. -- not sure what you mean here, because QSN strings are defined not to contain literal newlines. They're escaped like '\n'.
The invariant of QSN is: EVERY BYTE STRING, including those with newlines and nuls, can be represented on a single line. I guess I should put this in the documentation.
(2) A QSN decoder is very easy to write. For example, here's a ~6 line regex that validates all of QSN:
https://github.com/oilshell/oil/blob/master/qsn_/qsn.py#L498
True, you need like ~20 lines of code to decode it, but that's very easy too. You can also make a QSN decoder from a JSON string decoder. It's basically changing your tests and moving a few statements around.
(And yes, QSN is a regular language [1])
(3) Oil should grow [2] an awk-like dialect [3] that understands QSN and QTSV.
(4) Although you also don't have to decode it for it to be useful.
1. I can use wc -l on a stream of QSN strings
2. If I know the strings are QSN-encoded, I can search for NUL bytes with fgrep '\0'
It's basically like the UTF-8 philosophy. ASCII is valid utf-8. QSN lines are lines of text.
(5) base64 is bad for humans at the terminal, because it makes everything unreadable. QSN preserves all printable ASCII and unicode.
-----
I don't think it's going to solve all problems, but it will solve some. It's there if you need it. In typical Unix style, the solutions will be heterogeneous. You can absolutely use the NUL format in Oil, and it now has support for it.
[1] http://www.oilshell.org/blog/2020/07/eggex-theory.html
[2] That is, if I get help :)