Live data from Hacker News

Elvish – Scripting language and interactive shell

github.com

51–60 of 62 posts

Re: Elvish – Scripting language and interactive shell

#51
post #40

I've been using fish for many years now though I keep trying all these new shells. Ultimately I've found that for my interactive shell I just want something widely supported and with easy syntax for `if` and `for` loops for short multi-line commands. For anything longer than that I just reach for real Python using either the `sh` or `plumbum` package. I don't need the extra features very often, so I just run things i…

I'm also a fish fan and largely agree--but if you are forced to use Windows and you don't care for WSL, nushell is likely your best option. It's pretty good! Almost feels like Unix but you're in Windows. I don't think fish shell will be ported over to Windows anytime soon...

Re: Elvish – Scripting language and interactive shell

#52
post #48
post #34

Earlier quoted context omitted.

> does `bg` return the `fid` There's two kinds of process IDs in Murex: FID (function IDs) and PID (process IDs) Forking is expensive in POSIX and has a number of drawbacks such as the inability to share scoped variables without resorting to environmental variables. So a FID is basically a PID but managed inside the scope of Murex's runtime. You can manage FIDs in much the same way as you can manage PIDs, albeit usin…

Thanks for the answer! I think we may be talking past eachother a bit, but it's good to get confirmation on a lot of those! AFAICT named pipes have nothing to do with ssh tcp proxies, or at least that bit is tangential to the key point - the key point I was making is that ssh is running in the background while I'm running another command (with no relation between them, as far as the shell is concerned). You didn't an…

> You didn't answer my question about if `bg` returns a `fid` or not, and the documentation doesn't answer this either, nor did you say if those could be used in scripts... but it sounds like you're saying I have to parse `jobs` to get the `fid` of the command I just launched?

Everything in the documentation can be used in scripts.

`bg` doesn't write anything to stdout. The documentation does actually answer that if you look at the Usage section. In there it doesn't list . eg compare `bg` it to other builtins and you'll see the ones that do write to stdout have `-> ` in the usage. Also none of the examples show `bg` writing anything to stdout.

You wouldn't want `bg` to write the PID to stdout anyway because then it would be harder to use in scripts, because you'd then need to remember to pipe stdout to null or risk contaminating your script output with lots of random numbers.

However the good news is you can still grab the PID for any forked process using . eg

    » bg { sleep  99999; echo "Sleep finished" }
    » kill $MOD.sleep_pid
    Sleep finished
($MOD is required because bg creates a new scope so any local variables created in there wouldn't be visible outside of `bg` block. MOD sets the scope to module level (also supported are GLOBAL and ENV).

It's also worth reiterating my previous comment that `bg` blocks are not separate UNIX processes but instead just different threads of the existing Murex process. In fact all Murex builtins are executed as threads instead of processes. This is done both for performance reasons (UNIX processes are slow, threads are fast...relatively speaking) and because you cannot easily share data between processes. So if `bg` was a process, it would be impossible to grab the PID of `sleep` and then share it with the main process without then having to write complex and slow IPCs between each UNIX process.

> That is, the body of the while loop is executed asynchronously with the pre-pipe command.

Are you just talking about each command in the pipe executing concurrently? That's the default in Murex. for example the following would only work if each command is working as a stream:

    tail -f example.log | grep something | sed -e s/foo/bar/ | tee -a output.log
The loop example could be achieved via `foreach` too. eg

    while { true } { echo hi; sleep 1 } | foreach line { echo "$line 2" }
The only time `foreach` wouldn't stream is with data-types that aren't streamable, such as JSON arrays. But that's documented in the JSON docs (https://murex.rocks/types/json.html#tips-when-writing-json-i...)

> TBH I just looked at the Murex docs again and I'm lost. Where's the reference? There's the "user guide" which contains the "user guide" (again) and also the "beginners guide" which to me are synonyms, "cheat sheet" and "read more" which both appear to be a bunch of snippets (again synonyms?), "operators and tokens" which are a reference of just a subset of the language, etc etc. I couldn't find anything on loops, which I'd expect to be a section somewhere. Clicking on "read / write a named pipe" in "builtin commands" somehow teleports me to an identically named page in a completely different "operators and tokens" section.

I'd generally refer people to the language tour to begin with https://dev.murex.rocks/tour.html It's pretty visible on the landing page but sounds like we could do more to make it visible when elsewhere on the site. I can take that feedback and work to make to the tour more prominent in the menus too (the pages you described do also link to the language tour, but clearly not in a visible enough way)

> Also are named pipes always global? Why can't they be local variables like other structures?

The only reason for that is because named pipes were one of the original constructs in Murex. They came before modules, scoping, and so forth. It would be possible to make them scopeable too but careful thought needs to be made about how to achieve that in a backwards-compatible yet intuitive way. And there have been other requests raised by users and the other contributors that have taken priority.

> It's great you have so much documentation, but I think it actually turned me away - it seems bloated with every bit of information split into multiple pieces and scattered around. Simplifying, consolidating, and reorganizing it from the top might help.

We'd welcome some recommendations here. Organising documentation is really hard. Even more so when the people who organise it aren't the same people who are expected to depend upon it.

> Thanks! Yeah, I wasn't talking about arrays here, but if e.g. `x` in my example contains space-separated "these three words", will `command` receive 3 arguments or 1. In bash, `command` would get 3 arguments, if you don't invoke the variable quote hack. I just tried this out though, and it gets 1 argument, so great!

I know you weren't talking about arrays, but the next question people ask is "how do I now expand one variable to be multiple parameters?"

Re: Elvish – Scripting language and interactive shell

#54
post #23
post #17

I went down this route based on HN recommendations, with some people calling it stable well documented. There's TODOs all over the documentation! There's no background task tools for scripting, and in interactive use background tasks are barely supported - an issue about background tasks has people going roughly "nobody needs to do tasks in parallel, that was only important when people were working on mainframes". Th…

There’s also a lot of good design that’s gone into Elvish. And I don’t think it’s fair to call it “dead” when the maintainers for Elvish are active both on Github and here on HN too (probably other places too). However if you’re looking for an alternative then there’s: - Murex (disclaimer: I’m one of the maintainers) which does support background processes and has extensive documentation. https://murex.rocks - Nushel…

could you go into more detail how (and maybe why) murex and elvish differ?

also, i'd like to know more about how the job control works. that's one of the pain points in elvish, but both are written in go, so maybe there are some ideas that elvish could copy.

Re: Elvish – Scripting language and interactive shell

#55
post #54
post #23

Earlier quoted context omitted.

There’s also a lot of good design that’s gone into Elvish. And I don’t think it’s fair to call it “dead” when the maintainers for Elvish are active both on Github and here on HN too (probably other places too). However if you’re looking for an alternative then there’s: - Murex (disclaimer: I’m one of the maintainers) which does support background processes and has extensive documentation. https://murex.rocks - Nushel…

could you go into more detail how (and maybe why) murex and elvish differ? also, i'd like to know more about how the job control works. that's one of the pain points in elvish, but both are written in go, so maybe there are some ideas that elvish could copy.

Murex and Elivish share a lot of similar design goals. The author of Elvish has done a lot of great talks about his approach to Elvish development and you can tell a lot of care has gone into its design.

With Murex, I initially took a "let's just experiment until I find something that works" with no fear of writing ugly proof-of-concept code. This allowed me to build a lot of stuff very quickly and originally it was written in a self-hosted git repository to solve my own problems. But as the project evolved I realised there was some good stuff in there that's worth sharing. The downside to this approach is that there is some ugliness to its design that has lasted even to the latest version due to Murex's compatibility promise. However, the latest version of Murex does provide an internally versioned runtime, which means scripts can now pin to a specific version of Murex and not worry about gradual changes over time (even if "gradual over time" in this context literally means "years of compatibility" even before the versioned runtime.

This means that Murex and Elvish might feel like very different shells despite being conceptually quite similar.

I'm a little reluctant to give specific areas where the two shells diverge because both are under active development and thus moving targets. So what might be true today might not be true tomorrow. However, I will say the syntax for each does vary significantly despite being superficially similar.

As for job control, this was part of Murex's early design because it's a feature I used heavily at the time. So the concept of background and foreground processes are weaved throughout all of the core runtime. Like with Elvish, Murex doesn't create new UNIX processes for builtins. And with commands that are forked processes, Murex doesn't hand over complete ownership of the TTY to them so that Murex can still catch the signals. The reason for the latter is because Murex can then add additional hooks to job control, such as returning a list of open files any stopped processes have opened, and how far through reading those files it is. So Murex has needed to re-implement some of the job control logic that would normally be handled by the POSIX kernel. This does result in a lot of additional code, and thus places for things to go wrong. On balance, I think I made the right tradeoff for Murex. However if I were to write an entirely new shell from the ground up, I'd probably not do it this way again.

Re: Elvish – Scripting language and interactive shell

#56
post #55
post #54

Earlier quoted context omitted.

could you go into more detail how (and maybe why) murex and elvish differ? also, i'd like to know more about how the job control works. that's one of the pain points in elvish, but both are written in go, so maybe there are some ideas that elvish could copy.

Murex and Elivish share a lot of similar design goals. The author of Elvish has done a lot of great talks about his approach to Elvish development and you can tell a lot of care has gone into its design. With Murex, I initially took a "let's just experiment until I find something that works" with no fear of writing ugly proof-of-concept code. This allowed me to build a lot of stuff very quickly and originally it was…

i really like your approach to job control. my hope is that elvish can implement something similar. i am hopeful in that you already managed to overcome the challenges go introduces here, so the elvish devs can potentially take advantage of that.

Re: Elvish – Scripting language and interactive shell

#57
post #56
post #55

Earlier quoted context omitted.

Murex and Elivish share a lot of similar design goals. The author of Elvish has done a lot of great talks about his approach to Elvish development and you can tell a lot of care has gone into its design. With Murex, I initially took a "let's just experiment until I find something that works" with no fear of writing ugly proof-of-concept code. This allowed me to build a lot of stuff very quickly and originally it was…

i really like your approach to job control. my hope is that elvish can implement something similar. i am hopeful in that you already managed to overcome the challenges go introduces here, so the elvish devs can potentially take advantage of that.

The limitations here aren’t due to Go. You can define forked processes gpid and ctty, which are the two key pieces you need to define to “correctly” support job control.

And in fact Go actually makes it very easy to both catch job control signals raised by the kernel and set those aforementioned parameters when calling the fork syscall.

The real problem here is that we don’t actually want to POSIX compliant job control because that would mean builtins inside hot paths would perform significantly worse and we lose the ability to easily and efficiently share data between commands, such at type annotations, localised variables, etc.

The lack of type annotations is a particularly hard problem to solve and also the main reason to use an alternative shell like Murex or Elvish. In fact I’d say having type annotations work across commands is more important than job control.

So the end result is having to replicate a lot of what you would normally get for free in POSIX kernels, except this time running inside your shell. In places you’re basically writing kludge after kludge. But whenever I despair about the ugliness of the code I’ve written, I remind myself that this is all running on 40 year old emulation mechanical teletypes. So the whole stack is already one giant hot ball of kludges.

Re: Elvish – Scripting language and interactive shell

#58
post #57
post #56

Earlier quoted context omitted.

i really like your approach to job control. my hope is that elvish can implement something similar. i am hopeful in that you already managed to overcome the challenges go introduces here, so the elvish devs can potentially take advantage of that.

The limitations here aren’t due to Go. You can define forked processes gpid and ctty, which are the two key pieces you need to define to “correctly” support job control. And in fact Go actually makes it very easy to both catch job control signals raised by the kernel and set those aforementioned parameters when calling the fork syscall. The real problem here is that we don’t actually want to POSIX compliant job contr…

the challenges in go is a reference to a discussion in the elvish chat where one participant claimed that go's os.StartProces() API makes it impossible to implement unix job control with 100% fidelity.

i don't actually know what the issue there is, and maybe there is a way to avoid using os.StartProcess but the point is that murex is not implementing POSIX compliant job control. and that is one way to get around any issues that may exist.

and now having learned how murex handles job control i am happy that elvish avoided implementating POSIX job control so far because this allows rethinking how to approach this.

this is all running on 40 year old emulation mechanical teletypes

isn't that the real issue right there?

i have been wondering if it is not possible to get rid of that emulation layer and provide for a more rich way for programs to interact with the user.

we'll never be able to get rid of the emulation completely, but i wonder if the position in the stack can be moved.

right now it is:

    GUI
    GUI application that emulates a 40 yr old terminal
    shell
    programs running in the shell
how about:

    GUI
    GUI application for commandline programs that provides a rich interface
    modern shell that runs on that interface
    modern programs running in the shell
    or terminal emulation for legacy programs that need it
       legacy programs running with an emulation layer.
the emulation layer could be started by the shell as needed.

to get that emulation layer we only need to port something like tmux onto that new api. there is also a layer that implements job-control for shells that don't support it: https://github.com/yshui/job-security so this can be done without having to reimplement the emulation yet again

Re: Elvish – Scripting language and interactive shell

#59
post #58
post #57

Earlier quoted context omitted.

The limitations here aren’t due to Go. You can define forked processes gpid and ctty, which are the two key pieces you need to define to “correctly” support job control. And in fact Go actually makes it very easy to both catch job control signals raised by the kernel and set those aforementioned parameters when calling the fork syscall. The real problem here is that we don’t actually want to POSIX compliant job contr…

the challenges in go is a reference to a discussion in the elvish chat where one participant claimed that go's os.StartProces() API makes it impossible to implement unix job control with 100% fidelity . i don't actually know what the issue there is, and maybe there is a way to avoid using os.StartProcess but the point is that murex is not implementing POSIX compliant job control. and that is one way to get around any…

> the challenges in go is a reference to a discussion in the elvish chat where one participant claimed that go's os.StartProcess() API makes it impossible to implement unix job control with 100% fidelity.

That's not true. os.StartProcess() takes a pointer to https://pkg.go.dev/os#ProcAttr which then takes a pointer to https://pkg.go.dev/syscall#SysProcAttr

If I recall correctly, either Setctty or Foreground needs to be set to true (I forget which offhand, possibly Foreground, but browsing the StartProcess()'s source should reveal that.

I don't actually do that in Murex, because I want to add additional hooks to SIGSTSP and I can't do that if I hand ownership of the TTY to the child process.

https://github.com/lmorg/murex/blob/master/lang/exec_unix.go...

But that means that some tools like Helix then break job control in Murex because they don't think Murex supports job control (due to processes being marked non-traditionally). You can see higher up in that file above where I need to force Murex to take ownership of the TTY again as part of the process clean up. (line 28 onwards)

Processes invoked from a shell should also be part of a process group:

https://github.com/lmorg/murex/blob/master/lang/exec_unix.go...

You also need to set the shell to be a process session leader:

https://github.com/lmorg/murex/blob/master/shell/session/ses...

All of this is UNIX (inc Linux) specific so you can see compiler directives at the top of those files to exclude WASM, Windows, and Plan 9 builds. I don't even try to emulate job control on those platforms because it's too much effort to write, test, and maintain.

> i have been wondering if it is not possible to get rid of that emulation layer and provide for a more rich way for programs to interact with the user.

Funny enough, this is something I'm experimenting with my terminal emulator, though it's very alpha at the moment https://github.com/lmorg/Ttyphoon

> to get that emulation layer we only need to port something like tmux onto that new api

My terminal emulator does exactly this. It uses tmux control mode so that tmux manages the TTY sessions and the terminal emulator handles the rendering.

> https://github.com/yshui/job-security so this can be done without having to reimplement the emulation yet again

The problem with a 3rd party tool for job control is that it doesn't work with shell builtins. And the massive value add for shells like Murex and Elvish is their builtins. This is another reason why I didn't want POSIX job control in Murex: I wanted to keep my shell builtins powerful but also allow them to support job control in a way that feels native and transparent to the casual user.

Re: Elvish – Scripting language and interactive shell

#60
post #52
post #48

Earlier quoted context omitted.

Thanks for the answer! I think we may be talking past eachother a bit, but it's good to get confirmation on a lot of those! AFAICT named pipes have nothing to do with ssh tcp proxies, or at least that bit is tangential to the key point - the key point I was making is that ssh is running in the background while I'm running another command (with no relation between them, as far as the shell is concerned). You didn't an…

> You didn't answer my question about if `bg` returns a `fid` or not, and the documentation doesn't answer this either, nor did you say if those could be used in scripts... but it sounds like you're saying I have to parse `jobs` to get the `fid` of the command I just launched? Everything in the documentation can be used in scripts. `bg` doesn't write anything to stdout. The documentation does actually answer that if…

Awesome, thanks for the answers! It looks like Murex will be the first shell I try once I recharge my "rip everything up and start over" batteries again.

> I know you weren't talking about arrays, but the next question people ask is "how do I now expand one variable to be multiple parameters?"

I held back from asking that, but yes, exactly :P

Post reply on HN