Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Using Rust to write shell-script like tasks

github.com

Post reply on HN
Print view View original

61–70 of 83 posts

Re: Show HN: Using Rust to write shell-script like tasks

#61

Post by rhn_mk1 »

viraptor wrote: post #38
papaf wrote: post #26

Earlier quoted context omitted.

Since macros can execute arbitrary code, this makes me feel uneasy that the strings are just executed within a shell context, with all the appropriate, bug-prone, expansion done by the shell. Its a shame you didn't bother to look at the source code before criticizing. Someone put a lot of work into this library and its actually pretty cool. The package parses the code in the macros [1] and then calls 'std:Process::Co…

I don't think the parent said this is not parsed well, or at least I didn't read it that way. I share the feeling that you see that code and unless you know the implementation it's not clear what shell brokenness is carried over and what isn't. And which shell and version is being emulated. It's much easier to set expectations with a new syntax that's also easier to document than "what to expect of this macro".

Thank you, that's indeed what I meant.

The other, related issue is that reimplementing pieces of the shell DSL duplicates what can be done in the parent language.

Taking conditional return value as an example: "ls /nofile || true;"

In this case I don't really want to be given the option to use bash syntax for this. That would encourage the usage of shell idioms for "tricks" like control flow, which are another annoying part of the shell (I can never remember them). I would much prefer if there was a nice way to do that kind of things idiomatically in the parent language, and no other choice. E.g. to ignore the return value I would find it much nicer to be forced to do sth like this instead:

let _ = ls('nofile');

Re: Show HN: Using Rust to write shell-script like tasks

#62

Post by rustshellscript »

rhn_mk1 wrote: post #61
viraptor wrote: post #38

Earlier quoted context omitted.

I don't think the parent said this is not parsed well, or at least I didn't read it that way. I share the feeling that you see that code and unless you know the implementation it's not clear what shell brokenness is carried over and what isn't. And which shell and version is being emulated. It's much easier to set expectations with a new syntax that's also easier to document than "what to expect of this macro".

Thank you, that's indeed what I meant. The other, related issue is that reimplementing pieces of the shell DSL duplicates what can be done in the parent language. Taking conditional return value as an example: "ls /nofile || true;" In this case I don't really want to be given the option to use bash syntax for this. That would encourage the usage of shell idioms for "tricks" like control flow, which are another annoyi…

since run_cmd! and run_fun! are returning result type, you can always do let _ = run_cmd!(ls nofile); to ignore single command error.

The “xxx || true” is for ignoring error within a group of commands, which is also very common in sh “set -e” mode. Without it, the group of commands need to be divided into at least 3 parts to still capture all possible command errors. I probably need to document this part with more details.

Re: Show HN: Using Rust to write shell-script like tasks

#63

Post by rustshellscript »

IshKebab wrote: post #60
foota wrote: post #30

Earlier quoted context omitted.

This only seems to use the good parts of the shell, easy piping and redirection, while dropping the language for logic.

It also appears to use some bad parts, e.g. command line switches and unquoted arguments.

Unquoted argument is not an issue here, see some examples here: https://github.com/rust-shell-script/rust_cmd_lib/issues/10

Re: Show HN: Using Rust to write shell-script like tasks

#64

Post by rhn_mk1 »

rustshellscript wrote: post #62
rhn_mk1 wrote: post #61

Earlier quoted context omitted.

Thank you, that's indeed what I meant. The other, related issue is that reimplementing pieces of the shell DSL duplicates what can be done in the parent language. Taking conditional return value as an example: "ls /nofile || true;" In this case I don't really want to be given the option to use bash syntax for this. That would encourage the usage of shell idioms for "tricks" like control flow, which are another annoyi…

since run_cmd! and run_fun! are returning result type, you can always do let _ = run_cmd!(ls nofile); to ignore single command error. The “xxx || true” is for ignoring error within a group of commands, which is also very common in sh “set -e” mode. Without it, the group of commands need to be divided into at least 3 parts to still capture all possible command errors. I probably need to document this part with more de…

This ties in to what I wrote before: I prefer a philosophy where groups of commands are not written in the shell DSL, but are instead native statements (as much as possible), and the user is forced to use native control flow.

Documentation is not going to make me warm up to the idea, because I don't like having the choice to use the DSL so much.

With that in mind, perhaps I'm not the most valid person to provide criticism of this project ;)

Re: Show HN: Using Rust to write shell-script like tasks

#65

Post by kazinator »

intc wrote: post #17
kazinator wrote: post #6

Earlier quoted context omitted.

> unsafe Rust ... since you’re now forking external code Are you saying that Rust becomes unsafe because it used a C program as a subroutine? E.g. "tar xvf -" or whatever? What is the fix: rewrite tar, awk, scp and whatever else as Rust functions? That's a lot of work. I'm surprised that you're simultaneously overlooking what ought to be a more gaping problem: that every system call made by a Rust program is a trip t…

Could you please be more specific how it's a "gaping problem" that the underlying kernel is written in C? I think even you'd write a pure Rust kernel from scratch it would take a considerable time to achieve the same quality/performance ratio as we are currently witnessing with C based kernels (*BDS & Linux). It's so easy to throw these "radical claims". Yes? =)

The point is that if forking a process to invoke an external C program to run in another address space is "unsafe", directly calling into the OS (like making that fork call) should be considered "mega unsafe".

Re: Show HN: Using Rust to write shell-script like tasks

#66

Post by kazinator »

laumars wrote: post #8
kazinator wrote: post #6

Earlier quoted context omitted.

> unsafe Rust ... since you’re now forking external code Are you saying that Rust becomes unsafe because it used a C program as a subroutine? E.g. "tar xvf -" or whatever? What is the fix: rewrite tar, awk, scp and whatever else as Rust functions? That's a lot of work. I'm surprised that you're simultaneously overlooking what ought to be a more gaping problem: that every system call made by a Rust program is a trip t…

There’s a few problems with forking out: 1. Do those programs exist and what happened if they don’t? That behaviour is already understood in Bash, less so in random 3rd party Rust libraries. 2. Is ‘tar’ calling ./tar, /bin/tar or some other instance of tar? And how do you find out? (eg easy to check $PATH in Bash but does this library honour that? Easy to ‘which tar’ but is that going to be the same tar that this lib…

> Do those programs exist

If those programs don't exist, they are simply missing dependencies of the program.

In the shell, we might use the type command. Something similar could be integrated into this scripting system to detect whether some string corresponds to a command that can be found in the PATH.

> Is ‘tar’ calling ./tar, /bin/tar or some other instance of tar? And how do you find out?

PATH is actually used by low-level routines in POSIX, like execvp. If execvp is used as the basis for dispatching commands, then PATH is searched.

> A .rs file does not.

That's a fair point. Over the years, I have seen a fair share of C programs break because they were actually using system() or fork()/exec() to run programs that were missing or had some other problem.

I've also seen (and written myself) complex shell scripts that check for their dependencies up-front and complain if some are missing, which is a good idea, especially if not all execution paths use every dependency, or if an unexpected termination could occur after a lengthy process that the user will have to recover from and repeat.

It can also be loudly documented as part of the system requirements of the program. "This program relies on the utilities tar, awk and expect which are expected to be in the PATH. It was tested with GNU tar 1.29, GNU Awk 4.1.4 and Expect 5.45.4."

If we are packaging this program into a distro, we can express those dependencies in the packaging meta-data, so they are pulled in automatically. The package manager has to be conscientious and to understand that program's requirements.

Re: Show HN: Using Rust to write shell-script like tasks

#67

Post by Ar-Curunir »

darthrupert wrote: post #23

If this is a fun proof of concept, it's nice. If somebody uses this in an actual system, it's terrifying. edit oh, Rust is now a thing where even the bad ideas need to be praised without caveats. Gotcha

I mean, no one is suggesting that it be used in a critical system yet, so your suggestion is kinda unnecessary

Re: Show HN: Using Rust to write shell-script like tasks

#69

Post by kazinator »

laumars wrote: post #37
joshuamorton wrote: post #25

Earlier quoted context omitted.

> The point is that code doesn't belong in Rust to begin with! I'll reiterate: it is often safer to embed short snippets of bash into other languages than to invert control and call out to other languages from bash. By calling out to bash, you do the majority of your work in better languages. > No it isn't. Code auditing and vetting has been a thing for years. Say you have a CI pipeline that hooked into Shellcheck to…

> I'll reiterate: it is often safer to embed short snippets of bash into other languages than to invert control and call out to other languages from bash. By calling out to bash, you do the majority of your work in better languages. "It depends" is a better way of putting it. However the advantages of embedding Bash doesn't, in my opinion, make up for the problems it creates by obfuscating those calls. Putting Bash i…

I understand how that can go "bad". For instance, I worked with a really ugly connection manager written by Qualcomm. It was C++ code (object oriented with classes deriving from abstract bases and implementing virtual functions and all that).

At the bottom of the class hierarchy were methods that did their work with a hodge podge of system calls and invocations of external utilities like "ip" and "iptables" and whatever else.

The thing would react to netlink events in the kernel, paste together commands and pass them to system: not even using fork and exec to do it cleanly.

Just, eww.

Re: Show HN: Using Rust to write shell-script like tasks

#70

Post by dllthomas »

laumars wrote: post #57
oconnor663 wrote: post #49

I'm going to shamelessly plug my own library here: https://github.com/oconnor663/duct.rs I wanted to solve the same problem, originally in Python ( https://github.com/oconnor663/duct.py ). It's surprisingly annoying to do pipelines and redirections, compared to how easy they are to do in the shell. Lots of libraries try to address this, but most of them seem do it by emulating shell syntax within the host language, u…

‘cd’ is a shell builtin so you couldn’t use ‘cd’ in any of these solutions unless they then spawn a shell instance...and that worries me if you are because then you really might as well just have a separate .sh file and launch that instead (at least that is more auditable with tools like Shellcheck than any inlined code would be).

As I read the parent comment, the broad context is turning "shell-like behavior" into rust code, and the comment is choosing to talk about that projection by focusing on elements in the source and assuming that it's understood that they're really talking about the resulting image. You can't use the shell's cd, but you can call chdir and set the working directory - and hopefully you can do that for only part of your pipeline.

If they were in fact describing implementation, then I mostly agree - it's likely better to write shell directly than generate it, at least short of treating it seriously as a compilation target.

Post reply on HN