Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Marcel the Shell

marceltheshell.org

Post reply on HN
Print view View original

171–180 of 209 posts

Re: Marcel the Shell

#171

Post by feldrim »

jampekka wrote: post #154
9dev wrote: post #148

Earlier quoted context omitted.

Is that so? Because the way I see it, there are a bazillion different tools, plus decades of accrued knowledge only related to massaging data into text and back to structures. How many human brain cycles are wasted to data conversion on the Terminal? Text is a lousy abstraction. It may have worked well 50 years ago when computing was limited to researchers on American universities, but the digital world back then has…

When the alternative is to fire up some mess of an IDE with some nightmearish COM/.NET/ProtoBuf/whathaveyou overengineering marvels, I take grep and sed every time. Simple text-based structured data format (something like YAML) with possibility to embed binary would be the way to go IMHO. And of course one should be able to show graphics in the terminal. But we're not gonna get it. Open source tol is being rapidly en…

Well, that looks like a straw man fallacy. Assuming the only alternative is a full-blown enterprise IDE for a Unix shell is at least a strawman.

Re: Marcel the Shell

#172

Post by feldrim »

mgaunard wrote: post #116
larodi wrote: post #86

First of all - Linuxes and alike desperately need something like this shell becoming standard. And second - I always wonder how all these great minds who where throwing rubbish at Microsoft ever since... somehow forgot to mention that same company created important product called PowerShell like 20 years ago. And you know PowerShell is so good already you can easily drop it in any OS/X, Linux and it actually works. N…

sed and awk are simple tools; choosing an entirely different non-standard tool just because you can't learn them seems a bit overkill.

I believe you are missing the point. It's not about sed and awk. It's the string oriented approach to data requiring heavy usage of these tools for parsing the unstructured text.

The suggested alternative is the object oriented approach, just like PowerShell and the OP's project are based on.

Re: Marcel the Shell

#173

Post by t43562 »

To do the things I wanted to do I needed reduce and I think there's no documentation for it. It took me a while to understand the code:

    ls | map (f: f.size) | red +  # this adds all the file sizes together
one can also use lamdas:

    ls | map (f: f.size) | red (lambda acc,y: (y if not acc else acc+y))
(this does the same thing - adds up the file sizes)

BUT: how would I then divide the result by 1024? .... or assign the result to a variable?

There are a number of other functions one can use with "red" like concat, * (times), max, min, count, and, or and a few more.

At the moment I feel like I'm playing with a very fun toy that might give up on me at any moment and turn out to be limited. It obviously has limits but I'm really enjoying it.

Re: Marcel the Shell

#174

Post by celeritascelery »

zelphirkalt wrote: post #144

This looks a lot like an idea I am having for a shell written in GNU Guile. MI am still at the very beginning and POC phase. My ideas are: (1) making it possible to have any procedure that follows a certain interface to be usable as a "command" in the shell (2) have a fallback function, that gives you a procedure for a standard shell command like 'ls' (or reimplement your own!) (3) have piped commands automatically u…

This seems like it has some really cool ideas! This reminds me a little of eshell, which has some of the same features around allowing you to define shell functions in lisp. However it does run into issues moving between the lisp (sexp) and shell (string) boundaries.

Re: Marcel the Shell

#175

Post by mbreese »

jampekka wrote: post #150
mbreese wrote: post #141

Earlier quoted context omitted.

There are some terminals that can display inline images, but those features aren’t commonly used. Why? Text works. Text is the lowest common denominator for working with data. This is why terminals are still primarily text based. This is also why stdin/out/err are also commonly text based (when the user is expected to view it). Text is simple and easy to parse. Binary data exchange formats (or objects) come in and ou…

Text does work, but text data exchange formats come in and out of vogue just like binary. And when any structure is needed, we typically get some ad-hoc text format that doesn't interoperate. JSON has made things better here, but it's a bit of a verbose eyesore for humans. I think the lack a structured (text based) format is a definite shortcoming in UNIX. Whitespace delimited lists of strings do get you surprisingly…

While I think that ASCII including a specific record delimiter character (0x1E) that isn’t used is a missed opportunity, I actually think the lack of a set interchange format is a benefit.

No format will work for everyone and if there was one set format, we’d still have people complaining about how it doesn’t work for their usecase. We'd also have had to (finally) answer the question of how to send the schema with the data.

JSON (and XML and YAML and…) are all methods to try and adapt a text based data interchange format. And they all work roughly the same. You ultimately need some kind of processing library to full read them. But working with the data in any sort of structured context requires specialized tools (ex: jq).

Or you can still grep it, just like I do to manage my tab delimited data, and get part of the way.

I think the chaos of having no standard is kinda the point. Text is the lowest format that is easy for people to work with. We can visually parse it pretty easily, so the tools that support text can be pretty universal. Grep, sed, and awk can do a lot.

But to do anything more, you’re always going to need some kind of specialized parsing, even for tab delimited or CSV data. So, let the terminal work with generic text and be “good enough”.

As an example: you mentioned JSON and YAML and suggested a slimmed down YAML might be more powerful. I deal with columnar data with tens to hundreds of columns and thousands of lines. This wouldn’t work with either JSON or YAML… it’s too verbose. Instead I use (compressed) tab delimited text files for their ease of parsing in many languages. But those won’t work for many other use cases.

With data exchange, there is never going to be one right sized option for all needs. If you can accept that, then the “right” option is to choose none of them.

Re: Marcel the Shell

#176

Post by geophile »

mst wrote: post #164
geophile wrote: post #147

Earlier quoted context omitted.

I wrote Marcel, and its forerunner osh, also recently on HN. Osh actually preceded powershell, and in fact I don’t recall it receiving any acknowledgement from Microsoft . To be fair, I’m sure they never heard of it. The idea of piping objects has been rediscovered many times.

I don't see anything connecting the marcel site and the oilshell.org site, am I missing something (like, say, "this is a different osh") or did you just not get around to writing that up yet? (to be clear, this isn't a complaint, I'm just confused and asking for help becoming unconfused, I'm aware how lists of things to get around to writing up tend to be a grow-only datastructure for us all ;)

Sorry, osh as in object shell: https://github.com/geophile/osh.

Re: Marcel the Shell

#177

Post by geophile »

feldrim wrote: post #172
mgaunard wrote: post #116

Earlier quoted context omitted.

sed and awk are simple tools; choosing an entirely different non-standard tool just because you can't learn them seems a bit overkill.

I believe you are missing the point. It's not about sed and awk. It's the string oriented approach to data requiring heavy usage of these tools for parsing the unstructured text. The suggested alternative is the object oriented approach, just like PowerShell and the OP's project are based on.

Well there are two points. 1) Piping strings is dumb. 2) Multiple incompatible sublanguages has a long learning curve. How many people can just write complex awk/find scripts without checking the manual? Using marcel, and knowing python, I can do those complex things without RTFM.

Re: Marcel the Shell

#178

Post by geophile »

lars512 wrote: post #123
geophile wrote: post #14

Earlier quoted context omitted.

I wrote marcel. In my view, nushell and marcel are very similar. nushell has tabular output, which I haven't thought was all that useful. Marcel is python-based. The nushell guys have had to invent a lot of language. By basing marcel on python, I don't have to invent language. Any logic to be added is expressed in python. E.g, do a recursive listing of files and find those that changed in the last 3 days: ls -fr | se…

Marcel looks really interesting. As someone who's leaned into Nushell deeply, I can tell you that the tabular and hierarchical data output is the major selling point of Nushell. For me and my team, who work with data and metadata constantly, it's an incredible productivity boost. The major pains with Nushell are debugging pipelines that are too slow, having to learn the new language constructs, and process overhead f…

If you stick to marcel commands and python functions, then all the processing happens in a single Python process (with a small number of exceptions). Beside the process overhead, this approach also avoids serialization/deserialization costs, and even string copying costs. x | y | z just passes Python tuples from one command to another via function call.

Can you expand on your first point? Why are tabular and hierarchical output so important for your work?

Re: Marcel the Shell

#179

Post by geophile »

t43562 wrote: post #173

To do the things I wanted to do I needed reduce and I think there's no documentation for it. It took me a while to understand the code: ls | map (f: f.size) | red + # this adds all the file sizes together one can also use lamdas: ls | map (f: f.size) | red (lambda acc,y: (y if not acc else acc+y)) (this does the same thing - adds up the file sizes) BUT: how would I then divide the result by 1024? .... or assign the r…

Just continue the pipeline, e.g.

    ls | map (f: f.size) | red + | map (size: size / 1024)
You can also leave off "map", so:

    ls | (f: f.size) | red + | (size: size / 1024)
A pipeline produces a stream, so there's no clean way to get the single int result in a variable, but you can store the stream (containing the int) into a variable:

    ls | ... | (size: size / 1024) >$ x
Now x has the stream. To read back the stream and print it:

    x
To do something else with the stream:

    x
etc.

Re: Marcel the Shell

#180

Post by geophile »

zelphirkalt wrote: post #144

This looks a lot like an idea I am having for a shell written in GNU Guile. MI am still at the very beginning and POC phase. My ideas are: (1) making it possible to have any procedure that follows a certain interface to be usable as a "command" in the shell (2) have a fallback function, that gives you a procedure for a standard shell command like 'ls' (or reimplement your own!) (3) have piped commands automatically u…

A lot of this is already in marcel. A command run from the command line is a pipeline, e.g. list files recursively and output (path, filesize):

    ls -fr | (f: (f, f.size))
But now limit the command to .py files:

    ls -fr | select (f: f.suffix == '.py') | (f: (f, f.size))
Also, just look at files changed in the past day:

    ls -fr | select (f: f.suffix == '.py') | select (f: now() - f.mtime
Those selects can be abstracted. They can be turned into their own parameterized pipelines, and assigned to variables. Here is a pipeline to filter files by a given extension:

    ext = (| e: select (f: f.suffix == '.' + e) |)
And for files changed in the last d days:

    recent = (| d: select (f: now() - f.mtime
So the original pipeline becomes:

    ls -fr | ext py | recent 1 | (f: (f, f.size))
Post reply on HN