Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Shell script best practices, from a decade of scripting things

sharats.me

Post reply on HN
Print view View original

461–470 of 500 posts

Re: Shell script best practices, from a decade of scripting things

#461

Post by sargstuff »

maximus-decimus wrote: post #288
ducktective wrote: post #190

Earlier quoted context omitted.

>However, if you instead organize all your data in a format that's sympathetic to line-oriented processing on stdin-stdout, then shell will work with you instead of against. Not even that is necessary. Just use structured data formats like json. If you are consuming some API that is not json but still structured, use `rq` to convert it to json. Then use `jq` to slice and dice through the data. dmenu + fzf + jq + curl…

How do you use dmenu for your shell script? to launch it? to prompt the user for input while it's running? Do you have an example of a script you wrote?

WASM gawk with html as user input/output more flexible.

Re: Shell script best practices, from a decade of scripting things

#462

Post by sargstuff »

ndsipa_pomu wrote: post #362
palunon wrote: post #333

Earlier quoted context omitted.

> Which works not just to preserve the previous statement from internal inconsistency It doesn't. You now have 4 numbers.

0, 1, 3, 4 and infinity - there's four numbers in this industry. Five There's five numbers in this industry 0, 1, 3, 4, 5 and infinity Wait, I'll come in again

thought the industry was broken down in 8 bit increments (0, 8, 16, 32, 64, 128, etc)

log base 2 of 4 is only 16bits

Re: Shell script best practices, from a decade of scripting things

#463

Post by sargstuff »

usrbinbash wrote: post #264
xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

> What is the Shell paradigm? I would argue that it's line-oriented pipelines. Which python can do realitively well, by using the `subprocess` module. Here is an example including a https://porkmail.org/era/unix/award (useless use of cat) finding all title lines in README.md and uppercasing them with `tr` import subprocess as sp cat = sp.Popen( ["cat", "README.md"], stdout=sp.PIPE, ) grep = sp.Popen( ["grep", "#"], s…

shell was originally tied to job/programm processing.

Re: Shell script best practices, from a decade of scripting things

#464

Post by sargstuff »

lbhdc wrote: post #287
usrbinbash wrote: post #264

Earlier quoted context omitted.

> What is the Shell paradigm? I would argue that it's line-oriented pipelines. Which python can do realitively well, by using the `subprocess` module. Here is an example including a https://porkmail.org/era/unix/award (useless use of cat) finding all title lines in README.md and uppercasing them with `tr` import subprocess as sp cat = sp.Popen( ["cat", "README.md"], stdout=sp.PIPE, ) grep = sp.Popen( ["grep", "#"], s…

> But on the other side of that coin its alot easier in python to do a complex regular expression I am not sure I would agree. Sed fills this role quite nicely. cat README.md | grep # | tr '[:lower:] [:upper:]' | sed 's/something/something_else/'

one tool/command per 'concept' was a resource saving thing at one time.

sed is the thing that handles shell regular expressions for shellscripts.

Re: Shell script best practices, from a decade of scripting things

#465

Post by HenrikB »

> "15. Use shellcheck. Heed its warnings."

(Disclaimer: I'm one of the authors)

After falling in love with ShellCheck several years ago, with the help of another person, I made the ShellCheck REPL tool for Bash:

  https://github.com/HenrikBengtsson/shellcheck-repl
It runs ShellCheck on the commands you type at the Bash prompt as soon as you hit ENTER.

I found it to be an excellent way of learning about pit falls and best practices in Bash as you type, because it gives you instant feedback on possible mistakes. It won't execute the command until the ShellCheck issues are fixed, e.g. missing quotes, use of undefined variables, or incorrect array syntax.

It's designed to be flexible, e.g. you can configure ShellCheck rules to be ignored, and you can force executtion by adding two spaces at the end.

License: ISC (similar to MIT). Please help improve it by giving feedback, bug reports, feature requests, PRs, etc.

Re: Shell script best practices, from a decade of scripting things

#466

Post by sargstuff »

GoblinSlayer wrote: post #453
bitofhope wrote: post #197

Earlier quoted context omitted.

There are workloads where shell scripts are the so-called right tool for a job . All too often I see people writing scripts in "proper" languages and calling os.system() on every other line. Shell scripts are good for gluing programs together. It's fine to use them for that.

I wrote such a program, that runs other programs for heavy lifting but also parses text which you can't possibly do in bash.

bootloader, systemd, or init ?

Parsing text isn't anything fancy.

It's just knowing what the marker is for a word/item boundary.

For bash, that marker is defined in IFS

Re: Shell script best practices, from a decade of scripting things

#467

Post by sophacles »

GoblinSlayer wrote: post #451
sophacles wrote: post #416

Earlier quoted context omitted.

You say this as if it wasn't extremely common to find giant python monstrosities that can be replaced by a handful of lines of shell. TBF the shell code often is not just cleaner and easier to follow, but also faster. It's possible to use the wrong tool for the job in any language - including language choice itself. Dismissing a programming language because it's not shell and dismissing shell because it's not a prora…

Bash is a good tool if the script is short enough, but if you have to write more than 10 lines, then use a real language.

Nonsense. That's a terrible metric.

If i need to run 11 commands in a row, suddenly i need to make sure new tooling is installed in my instance and/or ship a binary?

What if that 11 lines is setting up some networking? Now i need to go write a 400 line go program to use netlink to accomplish the same task? Or should I condense that to 80 lines of go to shell out the commands that replicate the 11 lines of simple bash?

There are plenty of reasons to do this, I have done it more than once. None of those reasons are "crossed an arbitrary magic number of 'lines of shell'".

Re: Shell script best practices, from a decade of scripting things

#468

Post by sargstuff »

heresie-dabord wrote: post #189
HyperSane wrote: post #113

Earlier quoted context omitted.

sh and bash feel pretty primitive after learning PowerShell.

"Primitive" seems like the wrong word to apply. A modern Python programmer (a common example) might think that anything not OOP is archaic. But OOP is just one design pattern. And... it can easily lead to over-engineering and delight in the pattern per se. Bash, awk, grep et alii don't do OOP. But they are close to the data and are powerful. Compaints about the notation of these tools (e.g. "line noise") are becoming…

grep is a domain specific language.

Bash and gawk don't have direct in-language support for OOP, but can be used/abused to do things in an oop manner. aka without in-language support (abstractions/api), takes much more effort to do/understand OOP approach.

Re: Shell script best practices, from a decade of scripting things

#469

Post by pmarreck »

snidane wrote: post #204
xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

Shell and SQL make you 10x productive over any alternative. Nothing even comes close. I've seen people scrambling for 1 hours to write some data munging, then spend another 1 hour to run it through a thread pool to utilize those cores , while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes. What Python is to Java, Shell is to Python. It sp…

As far as control-R command history searching, really enjoying McFly https://github.com/cantino/mcfly

Re: Shell script best practices, from a decade of scripting things

#470

Post by pmarreck »

rcthompson wrote: post #326
hnlmorg wrote: post #227

Earlier quoted context omitted.

> getting the shell quoting hell right Shameless plug coming, it this has been a pain point for me too. I found the issue with quotes (in most languages, but particularly in Bash et al) is that the same character is used to close the quote as is used to open it.m. So in my own shell I added support to use parentheses as quotes in addition to the single and double quotation ASCII symbols. This then allows you to nest…

One of my favorite Perl features that has been disappointingly under-appropriated by other languages is quoting with q(...).

Ruby and Elixir both have features like this. Very sweet.

Elixir has sigils, which are useful for defining all kinds of literals easier, not just strings:

https://elixir-lang.org/getting-started/sigils.html#strings-...

You can also define your own. It's pretty great.

Post reply on HN