Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Four features that justify a new Unix shell

oilshell.org

Post reply on HN
Print view View original

171–180 of 185 posts

Re: Four features that justify a new Unix shell

#171

Post by esaym »

sergeykish wrote: post #155
esaym wrote: post #92

Earlier quoted context omitted.

I think the OP is mostly talking about "perl one liners" on the shell like this: https://catonmat.net/ftp/perl1line.txt It is not that hard either (unless perl is completely foreign to you). Ironically just today I wrote a python script (using concurrent.futures) to batch re-encode 10gb of mp3 podcasts into 3gb of opus files to save space. Before I deleted the old mp3's, I did a quick file count of mp3 and opus files…

It would be simpler (and match "everything after the last '/'") with perl -nle 'if( $_ =~ /([^\/]+)$/) { print "$1"}' I don't quite get how can stream of '* \.ogg' proceed through /[\w-]+.\w\w\w$/ filter. Notice space. And it allows ",ogg" (dot not escaped so it is not extension), thankfully find provides /\.ogg$/

Space was because HN uses it as italics and I don't know the escape. And yes there are cleaner ways, but the point was a real world example of having something not work out in pure shell but yet quickly barfing out something working with perl. And yes, looks like I didn't escape the dot, funny!

Re: Four features that justify a new Unix shell

#172

Post by chubot »

l0b0 wrote: post #144
chubot wrote: post #130

Earlier quoted context omitted.

A bunch of points: (1) Except when your tokens can contain newlines, which we're stuck with for the foreseeable future. -- not sure what you mean here, because QSN strings are defined not to contain literal newlines. They're escaped like '\n'. The invariant of QSN is: EVERY BYTE STRING, including those with newlines and nuls, can be represented on a single line. I guess I should put this in the documentation. (2) A Q…

> (1) Except when your tokens can contain newlines, which we're stuck with for the foreseeable future. -- not sure what you mean here, because QSN strings are defined not to contain literal newlines. They're escaped like '\n'. The tokens in question are filenames, which are (and will continue to be) allowed to contain newline characters. Sorry that wasn't clear. This was never meant to be a critique of QSN, I'm just…

If you've ever used JSON between processes, then QSN is the exact same idea. JSON strings are also encoded (but they can't represent byte strings).

There are probably people who never use JSON, and that's fine, but there are a lot more who do! The goal is absolutely for tools like grep to adopt QSN support. [1]

Another argument is that in networking framing, you have a few choices:

(1) delimiter-based (NUL bytes)

(2) escaping (QSN with \)

(3) length prefixed (netstrings)

So Oil supports #1 and #2 now. Oil doesn't support netstrings but it actually does make sense, and avoids encoding/decoding. Though QSN encoding can be made extremely fast, just like it's been done with JSON [2].

[1] GNU grep actually has the ASCII/binary detection problem that was brought up on HN awhlie ago. Honestly it would be worth making a "slow correct utf-8/QSN grep" that avoids this.

https://unix.stackexchange.com/questions/19907/what-makes-gr...

[2] https://github.com/simdjson/simdjson

Re: Four features that justify a new Unix shell

#173

Post by ywei3410 »

jhayward wrote: post #170
pmoriarty wrote: post #83

Earlier quoted context omitted.

I've actually started to transition my shell scripts to eLisp for better integration in to Emacs and eshell. As a Lisp, eLisp is not the greatest, but I'd still much rather use it than Python. I also don't want to sit and twiddle my thumbs while a Python script takes its sweet time in loading. Slow startup time is the kiss of death for most shell scripts.

You find Emacs to have a faster startup time than Python?

I'm not the original poster, but judging from the answer it looks as though they'd already have an Emacs session open and can execute it without any extra startup-time; in which case the answer is yes, because there's no startup time.

Re: Four features that justify a new Unix shell

#174

Post by mikepurvis »

d0mine wrote: post #169
mikepurvis wrote: post #167

Earlier quoted context omitted.

Right, and I do that, but it's an obvious hack; it causes a separate sh process to spawn, you open yourself to shell injection issues if you're not careful, teeing the pipe is trickier than it should be, etc etc.

you might have known this but the practice shows that It has to be repeated: there is no shell injection issues with literal string in your own code. YMMV but for those rare cases when I care about the shell injection, I just don't use shell and run the command directly in Python. Combination of shell one-liners and the main logic in Python works well in practice.

Oh yeah, for sure. I think it's just frustrating having that one more piece of mental overhead. Like, it should be that there's one sane way to do all this stuff in Python. Instead it's literal blobs of shell in some cases, and subprocess.STDXX pipes linked up in other cases, and maybe sometimes you use shlex to sanitize your arguments, or you give up on streaming and just use communicate() to get the whole result in memory at once. Blah.

Re: Four features that justify a new Unix shell

#175

Post by mixmastamyk »

ritchiea wrote: post #7
xilni wrote: post #4

I love the direction this wants to go in. So many of these are reasons why we've banned new shell scripts on our projects and instead make python scripts.

Are there ever good reasons to choose a shell script over a Python/Ruby/etc script?

Python is a bit clumsy by default when the primary purpose is to run other programs. But, it is easy to write a "run()" wrapper function that works more like a shell. There are modules like "sh" as well to smooth over the bumps.

Re: Four features that justify a new Unix shell

#176

Post by [deleted] »

mixmastamyk wrote: post #175
ritchiea wrote: post #7

Earlier quoted context omitted.

Are there ever good reasons to choose a shell script over a Python/Ruby/etc script?

Python is a bit clumsy by default when the primary purpose is to run other programs. But, it is easy to write a "run()" wrapper function that works more like a shell. There are modules like "sh" as well to smooth over the bumps.

[deleted]

Re: Four features that justify a new Unix shell

#177

Post by mixmastamyk »

pmoriarty wrote: post #84
zem wrote: post #81

Earlier quoted context omitted.

i find that ruby combines the best features of perl and python in that respect. makes shell-like scripts really convenient and easy while still providing the means to add structure.

Both Ruby and Python are too slow for me, and neither is Lispy enough.

Performance is not a factor in shell scripts where the majority of the work is done by C-programs and the script merely shuttles parameters to the desired places.

It's not as if bash were fast, mind you. Lisps already exist.

Re: Four features that justify a new Unix shell

#178

Post by d0mine »

mikepurvis wrote: post #174
d0mine wrote: post #169

Earlier quoted context omitted.

you might have known this but the practice shows that It has to be repeated: there is no shell injection issues with literal string in your own code. YMMV but for those rare cases when I care about the shell injection, I just don't use shell and run the command directly in Python. Combination of shell one-liners and the main logic in Python works well in practice.

Oh yeah, for sure. I think it's just frustrating having that one more piece of mental overhead. Like, it should be that there's one sane way to do all this stuff in Python. Instead it's literal blobs of shell in some cases, and subprocess.STDXX pipes linked up in other cases, and maybe sometimes you use shlex to sanitize your arguments, or you give up on streaming and just use communicate() to get the whole result in…

I don’t see it as an overhead, I see using shell syntax as just another DSL like regex.

There are many different use cases related to running processes—it is natural that different solutions may be preferred for different cases (check_{call,output}, with/out shell, run, Popen, pty/PIPE, threads/Asunción—all may be useful. And it is just stdlib ).

Re: Four features that justify a new Unix shell

#179

Post by l0b0 »

chubot wrote: post #172
l0b0 wrote: post #144

Earlier quoted context omitted.

> (1) Except when your tokens can contain newlines, which we're stuck with for the foreseeable future. -- not sure what you mean here, because QSN strings are defined not to contain literal newlines. They're escaped like '\n'. The tokens in question are filenames, which are (and will continue to be) allowed to contain newline characters. Sorry that wasn't clear. This was never meant to be a critique of QSN, I'm just…

If you've ever used JSON between processes, then QSN is the exact same idea. JSON strings are also encoded (but they can't represent byte strings). There are probably people who never use JSON, and that's fine, but there are a lot more who do! The goal is absolutely for tools like grep to adopt QSN support. [1] Another argument is that in networking framing, you have a few choices: (1) delimiter-based (NUL bytes) (2)…

> If you've ever used JSON between processes, then QSN is the exact same idea. JSON strings are also encoded (but they can't represent byte strings).

I don't see how they are anything like the same idea. JSON is an object serialization format. These objects may contain strings, and the encoding of those strings unfortunately is not binary-complete. But most importantly objects have structure. QSN is a binary string encoding format, and has no structure outside of the sequence of bytes. In any case I'm not sure why you keep bringing up JSON.

On the other hand, if it wasn't for the fact that JSON is so strongly tied to JavaScript QSN might've been a good string encoding format for it.

> There are probably people who never use JSON, and that's fine, but there are a lot more who do! The goal is absolutely for tools like grep to adopt QSN support. [1]

I know what the words mean (and I've been using JSON and grep for years), but I don't understand what that sentence means.

> Another argument is that in networking framing, you have a few choices:

I don't know what "networking framing" is. If you mean actual network packet structure, I don't expect you'll be able to convince a single network engineer that QSN is a better choice than netstrings.

In any case, I'm going to have to shut down this thread now. I don't expect anyone else is reading this, I'm not learning anything new, and I can't convince you that QSN is a bad idea for IPC.

Re: Four features that justify a new Unix shell

#180

Post by gray_-_wolf »

esaym wrote: post #92
FreshFries wrote: post #71

Earlier quoted context omitted.

There is nothing in the (IT) world I fear more then... Perl projects. The shear madness it will leash upon you when the dependencies fail is maddening. Never will I voluntarily touch anything written in Perl ever again in my live. I've seen the power of Perl, the beauty and elegance of it, but everything was ruined by the absolute shit show when it comes to its packet managers. /rant.

I think the OP is mostly talking about "perl one liners" on the shell like this: https://catonmat.net/ftp/perl1line.txt It is not that hard either (unless perl is completely foreign to you). Ironically just today I wrote a python script (using concurrent.futures) to batch re-encode 10gb of mp3 podcasts into 3gb of opus files to save space. Before I deleted the old mp3's, I did a quick file count of mp3 and opus files…

Or, in actual posix compatible way:

    find /tmp/test -type f -iname '*.ogg' -exec sh -c 'for f; do printf "%s\n" "${f##*/}"; done' - {} +
Post reply on HN