Cicada – Unix shell written in Rust • Hacker News Board

Re: Cicada – Unix shell written in Rust

#111

Post by OvidNaso » Sat, Jul 01, 2017, 5:44 PM UTC

brandur wrote: post #84

zokier wrote: post #82

Earlier quoted context omitted.

What sort of features do you feel that shell needs to support structured data? I've toyed with this idea, and in my view the shell itself has relatively minor role here. What is needed are utilities that produce and process structured data, and a terminal that can display it. But shell, it just glues the pieces together and doesn't really need to be aware of the data and the structure of it.

Here's a very simple example that I'd consider the holy grail. Say I want run `ls -lh` on a directory: -rw-r--r-- 1 brandur staff 6.5K Mar 25 15:37 Makefile -rw-r--r-- 1 brandur staff 63B Jul 10 2016 Procfile -rw-r--r-- 1 brandur staff 1.6K Mar 11 07:20 README.md drwxr-xr-x 4 brandur staff 136B Oct 16 2016 assets/ drwxr-xr-x 4 brandur staff 136B Jul 15 2016 atom/ drwxr-xr-x 4 brandur staff 136B Apr 26 2016 cmd/ I'm i…

'ls -lh | column --table --output-separator "," | cut --delimiter="," --fields=6-8 --output-delimiter=" " | sort --month-sort'

I statred this to show how easy it was, but it quickly became pretty hacky.

Re: Cicada – Unix shell written in Rust

#112

Post by mos_basik » Sat, Jul 01, 2017, 5:51 PM UTC

vog wrote: post #12

zdkl wrote: post #7

Earlier quoted context omitted.

My structured objects aren't your structured objects. In that use case why not use a dedicated program (like say ae python/js/clojure interpreter) to handle more complex pipes and keep the shell level primitves... Well, primitive. Text is compatible with all systems, past and present, and can be used to model more complex objects. Let's not add features to the core on a "why not" please.

The so-called "plain text formats" are also just representations of complex objects. Often these formats are ad-hoc and hard to parse correctly. Multiple text files in a file system hierarchy (e.g. /etc) are also just nested structures. So in principle, treating those as complex object structures is the right way to go. Also, I believe this is the idea behind D-Bus and similar modern Unix developments. However, what'…

On phone so unsure if it's been mentioned in the thread already, but I've been enjoying learning a js library JSONata [0] after finding it included in Node-RED.

I remember finding jq a few weeks ago and thinking "wow, this will probably come in handy for a specific kind of situation" and filing it mentally for later use, but I haven't used it for anything yet so I'm not super familiar with the extent of it's features.

I have been using a lot of JSONata one liners to replace several procedural functions that were doing data transforms on JSON objects, and I'm very impressed. It's a querying library but it's Turing complete - it has lambdas, it can save references to data and functions as variables, etc.

It also seems relatively new/unknown; I've found hardly any blogs or forums mentioning it. The developer is active - he fixed a bug report I submitted in less then a day.

I'd love to have that kind of functionality in a CLI tool. Maybe jq is equally powerful, I don't know.

I haven't had time to run any performance analysis on JSONata and haven't found anyone else online who's done any yet. I'm very curious how its queries compare to efficiently implemented procedural approaches.

0: http://jsonata.org/

Re: Cicada – Unix shell written in Rust

#113

Post by skywhopper » Sat, Jul 01, 2017, 6:01 PM UTC

brandur wrote: post #3

Amazing work! Between this, Alacritty [1], and coreutils [2], we're getting pretty close to a plausible all-Rust CLI stack. While Cicada is pretty clearly modeled on the "old" generation of shells (sh, Bash, Zsh, etc.), one has to wonder what a more modern shell might look to address some of the problems of its predecessors like pipes that are essentially text-only, poor composability (see `cut` or `xargs`), and terr…

I certainly did like the idea behind Powershell's structured-object pipelines, but it relies on that "COM baggage" and "elaborate syntax" that you say are its downsides. I don't think you can achieve the goal behind Powershell of a universal standard binary object format without an elaborate infrastructure like COM and without making a bunch of assumptions and requirements on how those objects must look and behave, which will vastly limit the usefulness of the pipeline.

All of that said... nothing is stopping you from piping binary objects between tools in Unix shells. In fact, it works great, and I'm curious what functionality you are looking for that doesn't exist. The standard tools like cut and grep don't work great with binary data, but they aren't meant to. There are format-specific tools aplenty, and stuff like bbe exist for generic work. And for structured text, tools like `jq` are phenomenal.

Re: Cicada – Unix shell written in Rust

#114

Post by mos_basik » Sat, Jul 01, 2017, 6:06 PM UTC

tcbawo wrote: post #109

erk__ wrote: post #96

Earlier quoted context omitted.

You can also do it in PowerShell, which is on linux now. PS C:\Users\erk> Get-ChildItem | Sort-Object LastWriteTime | Select-Object -first 1 Directory: C:\Users\erk Mode LastWriteTime Length Name ---- ------------- ------ ---- d----- 17-06-2016 16:08 Tracing

Part of me thinks that PowerShell has seen less adoption in the Unix world because of the prevalence of camel case. The few times I used PowerShell, the discoverability of what I could do was low and the verbosity of examples was high. It took me too long to do my most common tasks.

I've seen several people who like PS say its discoverability is actually pretty great compared to Unix shells - for instance, the convention for cmdlet names is always "verb-noun" ("action-object") and (maybe, don't quote me on this) the set of blessed verbs people are supposed to use in cmdlet names is fairly small. So, when you want to do something, you try the verbs that seem closest to what you want to do and the names of the (usually application specific) objects you're working with and that gets you quite a long way down the road.

Also I think PS commands aren't case sensitive, though for readability one would probably want to keep the camelcase anyway.

Re: Cicada – Unix shell written in Rust

#115

Post by elcritch » Sat, Jul 01, 2017, 6:07 PM UTC

mos_basik wrote: post #112

vog wrote: post #12

Earlier quoted context omitted.

The so-called "plain text formats" are also just representations of complex objects. Often these formats are ad-hoc and hard to parse correctly. Multiple text files in a file system hierarchy (e.g. /etc) are also just nested structures. So in principle, treating those as complex object structures is the right way to go. Also, I believe this is the idea behind D-Bus and similar modern Unix developments. However, what'…

On phone so unsure if it's been mentioned in the thread already, but I've been enjoying learning a js library JSONata [0] after finding it included in Node-RED. I remember finding jq a few weeks ago and thinking "wow, this will probably come in handy for a specific kind of situation" and filing it mentally for later use, but I haven't used it for anything yet so I'm not super familiar with the extent of it's features…

Thanks! Will have to try out `jsonata`, as `jq` never clicked for me (too complex). Alternately I've been using a node.js program `json` [1] which has a basic but straightforward cli which covers 95% of my daily needs. I believe I found after trying to figure out the tool Joyent.com uses in there SmartOS sysadmin setup. Not sure if it's the exact same tool, but it's very similar.

As an example, basic manipulation is just:

`echo '{"age":10}' | json -e 'this.age++' #=> {"age": 11}`

1: https://github.com/trentm/json

Re: Cicada – Unix shell written in Rust

#116

Post by skywhopper » Sat, Jul 01, 2017, 6:09 PM UTC

vog wrote: post #12

zdkl wrote: post #7

Earlier quoted context omitted.

My structured objects aren't your structured objects. In that use case why not use a dedicated program (like say ae python/js/clojure interpreter) to handle more complex pipes and keep the shell level primitves... Well, primitive. Text is compatible with all systems, past and present, and can be used to model more complex objects. Let's not add features to the core on a "why not" please.

The so-called "plain text formats" are also just representations of complex objects. Often these formats are ad-hoc and hard to parse correctly. Multiple text files in a file system hierarchy (e.g. /etc) are also just nested structures. So in principle, treating those as complex object structures is the right way to go. Also, I believe this is the idea behind D-Bus and similar modern Unix developments. However, what'…

The brilliant thing about the status quo is that it already does accomodate both line-based and structured pipelines. There's no reason we should expect to use the same set of commands for both purposes. In fact, we most definitely shouldn't. A tool like jq is a great example of this: it works great for JSON, and it doesn't attempt to do more. When we want to work with JSON in a pipeline, we pull out jq. When we want to work with XML, we pull out xmlstarlet or the like. When we want to work with delimited columnar data, we can use awk or cut. I'm failing to see what's missing from an architectural point of view.

Re: Cicada – Unix shell written in Rust

#117

Post by erk__ » Sat, Jul 01, 2017, 6:41 PM UTC

mos_basik wrote: post #114

tcbawo wrote: post #109

Earlier quoted context omitted.

Part of me thinks that PowerShell has seen less adoption in the Unix world because of the prevalence of camel case. The few times I used PowerShell, the discoverability of what I could do was low and the verbosity of examples was high. It took me too long to do my most common tasks.

I've seen several people who like PS say its discoverability is actually pretty great compared to Unix shells - for instance, the convention for cmdlet names is always "verb-noun" ("action-object") and (maybe, don't quote me on this) the set of blessed verbs people are supposed to use in cmdlet names is fairly small. So, when you want to do something, you try the verbs that seem closest to what you want to do and the…

Yeah PowerShell is not case sensitive, and it also have a lot of unix-like aliases you can use, Another way to write the command of post is:

  dir | sort LastWriteTime | select -first 1

It gives the same output.

Re: Cicada – Unix shell written in Rust

#118

Post by eridius » Sat, Jul 01, 2017, 6:50 PM UTC

floatboth wrote: post #54

cies wrote: post #44

There's also this project: https://github.com/redox-os/ion It's intially build with RedoxOS (in Rust as well) underneath, but now also runs on Linux. Seems to be a bit more mature. I wonder what sets these two projects apart, and/or if the devs of Casada know of Ion's existence.

Cicada seems to be more POSIX-y, while Ion "it is not, nor will it ever be, compliant with POSIX". What they have in common though… is that job control (Ctrl+Z) is not implemented yet :(

I've been giving thought lately to writing my own shell, because there's stuff I want to fix that no current shell does right, but job control is one of the big reasons why I haven't actually started writing code yet (namely, I haven't yet figured out how to reconcile the stuff I want to do with the need to let the user suspend or background tasks including shell scripts).

Re: Cicada – Unix shell written in Rust

#119

Post by naasking » Sat, Jul 01, 2017, 6:51 PM UTC

IshKebab wrote: post #63

ue_ wrote: post #60

Earlier quoted context omitted.

Why shouldn't it? The shell is arguably the most convenient way to interact with your system's programs, seconded only by something like Perl. That level of interaction ought to mean you can write functions such as for common tasks. Or should I now have to use Perl for that?

Because fragile shell scripts have been an endless source of security vulnerabilities and bugs. I would probably use Python, Go or Powershell.

Shells languages arguably should be usable programming languages. So start with a good, expressive one add add scripting as a library:

http://users.eecs.northwestern.edu/~jesse/pubs/caml-shcaml/

Incidentally, that would also dramatically decreased security and maintainability problems.

Re: Cicada – Unix shell written in Rust

#120

Post by zokier » Sat, Jul 01, 2017, 6:51 PM UTC

brandur wrote: post #84

zokier wrote: post #82

Earlier quoted context omitted.

What sort of features do you feel that shell needs to support structured data? I've toyed with this idea, and in my view the shell itself has relatively minor role here. What is needed are utilities that produce and process structured data, and a terminal that can display it. But shell, it just glues the pieces together and doesn't really need to be aware of the data and the structure of it.

Here's a very simple example that I'd consider the holy grail. Say I want run `ls -lh` on a directory: -rw-r--r-- 1 brandur staff 6.5K Mar 25 15:37 Makefile -rw-r--r-- 1 brandur staff 63B Jul 10 2016 Procfile -rw-r--r-- 1 brandur staff 1.6K Mar 11 07:20 README.md drwxr-xr-x 4 brandur staff 136B Oct 16 2016 assets/ drwxr-xr-x 4 brandur staff 136B Jul 15 2016 atom/ drwxr-xr-x 4 brandur staff 136B Apr 26 2016 cmd/ I'm i…

But if we have a program that produces structured data ("ls"/"get-childitem") and a program that similarly consumes structured data ("select-column"), then I'm not sure what sort of "smartness" you would need from the pipe in between? Why would the shell need to understand that the data is somehow more than just stream of bytes?

As far as pretty-printing the final output to the user, in my opinion that is something that is better handled at terminal level instead of the shell.

Of course these are just comments based on how I imagine a reasonable object shell would function, and I really would like to hear more views on this subject because it is very likely that I might have overlooked something essential.