Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Four features that justify a new Unix shell

oilshell.org

Post reply on HN
Print view View original

111–120 of 185 posts

Re: Four features that justify a new Unix shell

#111

Post by chubot »

CameronNemo wrote: post #97
chubot wrote: post #90

Earlier quoted context omitted.

(author here) I do use Python. In fact I wrote something like 30K lines of Python for Oil. The question isn't shell OR Python. My shell scripts call Python scripts, many of white I wrote myself. That is working as intended. They also call C programs, C++ programs and R programs. And put JavaScript programs in various places. ---- I guess this is a huge misconception about shell that I have to write a blog post about.…

>When you program in shell, gcc, git, pip, npm, markdown, rsync, diff, perf, strace, etc. are part of your "standard library". I don't think this is a good characterization at all... Shell does not have git functionality (for example) built in. That is a dependency just like it would be for a python or rust project.

What I mean is that shell speaks paths, pipes, and processes natively, and you can creatively use tools that are "just there". Example: http://www.oilshell.org/blog/2017/09/19.html

It's an analogy, not a precise statement. It could be made more precise by using Oil as the center of a container-based "semi-distro", i.e. a distro that does everything that's not hardware related.

I guess a little like the complement to what CoreOS was doing (or is?). (This is a project I've been thinking about for awhile; anyone should feel free to contact me if they've done something like that or have ideas. It's related to the dev tools problems described in the blog post.)

Re: Four features that justify a new Unix shell

#112

Post by AareyBaba »

esaym wrote: post #110
high_priest wrote: post #106

Earlier quoted context omitted.

Why am I not surprised that the PERL part could be solved with bash in the same space. Why are you so eager to use lang on top of lang, in such simple cases?

Feel free to demonstrate. Take a long path with nasty characters and give me the ending file name. I want "track1.mp3" returned from: /podcast/with/nasty'--c h a r$/and/stuff/track1.mp3

$ basename "/podcast/with/nasty'--c h a r$/and/stuff/track1.mp3"

track1.mp3

Re: Four features that justify a new Unix shell

#113

Post by bscphil »

mikepurvis wrote: post #102
jonahx wrote: post #67

Earlier quoted context omitted.

The author addresses this argument: > However, Python and Ruby aren't good shell replacements in general. Shell is a domain-specific language for dealing with concurrent processes and the file system. But Python and Ruby have too much abstraction over these concepts, sometimes in the name of portability (e.g. to Windows). They hide what's really going on. From: http://www.oilshell.org/blog/2018/01/28.html#i-dont-unde…

Even something that should be as simple as piping output from one process to another in Python is a nightmare, and fraught with opportunities to shoot yourself in the foot with buffering deadlocks and other nonsense. The consequence of this is that many of my projects which require these overlapping domains (think stuff like build automation) ends up being the two interleaved— either an outer shell script that calls…

> Even something that should be as simple as piping output from one process to another in Python is a nightmare, and fraught with opportunities to shoot yourself in the foot with buffering deadlocks and other nonsense.

This hits the nail on the head for me. What I'd like is something bash-like to handle running programs, pipes, tab-complete for file names, and so on, and something like Python syntax for control flow and strings (quoting / escaping), and access to something powerful like Python's standard math libraries (and ability to import other stuff like requests). I don't know how you'd roll all that into a single shell, but as close as you can get is what I'd like to see in a new shell.

(I've briefly looked at Oil before, but it has seemed a bit complicated to merit trying to switch full time at this point. I'm definitely following it for future developments though.)

Edit: just found Xonsh from another user's comment, which seems almost exactly what I'm looking for, if a bit hacked together at first glance. Going to try that out.

Re: Four features that justify a new Unix shell

#114

Post by zimpenfish »

esaym wrote: post #92
FreshFries wrote: post #71

Earlier quoted context omitted.

There is nothing in the (IT) world I fear more then... Perl projects. The shear madness it will leash upon you when the dependencies fail is maddening. Never will I voluntarily touch anything written in Perl ever again in my live. I've seen the power of Perl, the beauty and elegance of it, but everything was ruined by the absolute shit show when it comes to its packet managers. /rant.

I think the OP is mostly talking about "perl one liners" on the shell like this: https://catonmat.net/ftp/perl1line.txt It is not that hard either (unless perl is completely foreign to you). Ironically just today I wrote a python script (using concurrent.futures) to batch re-encode 10gb of mp3 podcasts into 3gb of opus files to save space. Before I deleted the old mp3's, I did a quick file count of mp3 and opus files…

If I've not wildly misunderstood what you're doing with that line (and apologies if I have), could you not just stick `-printf '%f\n'` on the end of the `find` command?

(By definition, since you're looking for `.ogg` files, everything `find` finds will have a path that ends in a three letter extension.)

Re: Four features that justify a new Unix shell

#115

Post by thezilch »

esaym wrote: post #110
high_priest wrote: post #106

Earlier quoted context omitted.

Why am I not surprised that the PERL part could be solved with bash in the same space. Why are you so eager to use lang on top of lang, in such simple cases?

Feel free to demonstrate. Take a long path with nasty characters and give me the ending file name. I want "track1.mp3" returned from: /podcast/with/nasty'--c h a r$/and/stuff/track1.mp3

  $ path="/podcast/with/nasty'--c h a r$/and/stuff/track1.mp3"
  $ echo "${path##*/}"
  track1.mp3
  $ basename "${path}"
  track1.mp3
  $ echo "${path}" | awk -F/ '{print $NF}'
  track1.mp3

Re: Four features that justify a new Unix shell

#116

Post by acdha »

jonahx wrote: post #67
pierrebai wrote: post #60

I'm sorry to say to for me, shell scripting has lost. All my scripts are written in Python, which a thin shell script wrapper to launch it. Look at Oil. It tries hard to be bash compatible with optional buy-in to extension. Just use a superior language and give up on shell scripts, that's my advice.

The author addresses this argument: > However, Python and Ruby aren't good shell replacements in general. Shell is a domain-specific language for dealing with concurrent processes and the file system. But Python and Ruby have too much abstraction over these concepts, sometimes in the name of portability (e.g. to Windows). They hide what's really going on. From: http://www.oilshell.org/blog/2018/01/28.html#i-dont-unde…

That’s not a very convincing post because it’s a bit too shallow:

> I encountered a nice blog post, Replacing Shell Scripts with Python, which, in my opinion, inadvertently proves the opposite point. The Python version is longer and has more dependencies. In other words, it's more difficult to write and maintain.

The Python version uses only the standard library, so it doesn’t make sense to worry about dependencies if you aren’t counting every program your shell scripts call similarly. Having just one thing providing a consistent baseline beats learning that your script needs to upgrade RHEL to get a feature which would make something safer or easier.

Similarly, the Python script is longer because it does better work and consistent, useful error handling. That adds lines of code but it’s usability and correctness, not overhead.

It’s similarly incorrect to treat line count as a proxy for difficulty, as anyone who’s ever had to deal with quoting or data structures in shell scripts knows. The use of pathlib or os.path sometimes gets grumbles for extra characters but I’ve found it nearly inevitable that the grumblers will have something fail or destroy data because they hit a filename or argument with a space or special character in it. On one notable occasion, that resulted in an `rm -rf something something /`. (If you’re writing a shell script, shellcheck is mandatory)

The best criticism I’d make is that it could be easier to replace a shell pipeline. Python’s subprocess makes that pretty easy but you still need a minimum of 3 calls per command plus whatever you need to actually do with the output. Since subprocess.{run,check_output} handles most of my needs this isn’t a big deal and there are excellent modules if you don’t mind dependencies.

Re: Four features that justify a new Unix shell

#117

Post by zimpenfish »

esaym wrote: post #110
high_priest wrote: post #106

Earlier quoted context omitted.

Why am I not surprised that the PERL part could be solved with bash in the same space. Why are you so eager to use lang on top of lang, in such simple cases?

Feel free to demonstrate. Take a long path with nasty characters and give me the ending file name. I want "track1.mp3" returned from: /podcast/with/nasty'--c h a r$/and/stuff/track1.mp3

If you have that in a variable, which you presumably would if you're reading them one at a time, you can use "remove longest matching prefix" substitution.

    $ i="/podcast/with/nasty'--c h a r$/and/stuff/track1.mp3"
    $ echo "${i##*/}"
    track1.mp3
[Edit: 'thezilch beat me too it by a couple of minutes]

Re: Four features that justify a new Unix shell

#118

Post by l0b0 »

e12e wrote: post #46
l0b0 wrote: post #19

There are some good ideas in here, but `--qsn` is described[1] like this: > Print filenames ONE PER LINE. If a name contains a newline or other special char, it's QSN-encoded like 'multi-line \n name with NUL \0 byte' This is a bad idea. There is a solution which already works. Just terminate every filename with NUL and you're done. Trying to make something "sorta kinda human readable" is a mistake when dealing with…

I think having both is a good idea. Sometimes maybe you get your list of path names from a Here-document, for example.

For sure we need both human- and machine-readable stream formats, but I don't really see how QSN has any advantage in that space over `printf '%q\n'` and NUL-terminated strings, respectively.

Re: Four features that justify a new Unix shell

#119

Post by ZeteticElench »

asiachick wrote: post #101

How about Powershell https://github.com/PowerShell/PowerShell I can't call myself a fan of powershell but if everyone switched I'd get used to it. sh and bash both seem like they should die in a fire. They are full of foot guns that end up costing millions in breaches and lost data. The space think even bit Apple back in the day, they had some OS upgrade script that ended up deleting your entire hard drive if there w…

PowerShell is pretty damn awesome once you get the hang of it. The way objects are passed via pipe allows for really cool things like get-foo | where {$_.bar -eq 1} and you can serialize the output of most commands with get-foo | convertto-json. I use this feature all the time. Get-ADObject | convertto-json will give you a dump of your entire AD.

Re: Four features that justify a new Unix shell

#120

Post by tingletech »

FreshFries wrote: post #71
R0b0t1 wrote: post #66

Earlier quoted context omitted.

There is a better option and its name is Perl. Perl mixes the ins of shell scripts like easy argument and output passing with complex and easy to use control structures. You could use Python, but having been forced to use Perl extensively Perl is the superior choice for a complex shellscript-like workload. There's less boilerplate in Perl. There's a couple Python projects that come close like Fabric, but Perl was lit…

There is nothing in the (IT) world I fear more then... Perl projects. The shear madness it will leash upon you when the dependencies fail is maddening. Never will I voluntarily touch anything written in Perl ever again in my live. I've seen the power of Perl, the beauty and elegance of it, but everything was ruined by the absolute shit show when it comes to its packet managers. /rant.

I have 20 year old Perl code I run in production. I had to re-install it our move to Amazon Linux 2. My 10 year old install process for the cpanm setup still worked. My 10 year old Python had to be moved to python 3; but the 20 year old Perl still runs
Post reply on HN