Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Shell script best practices, from a decade of scripting things

sharats.me

Post reply on HN
Print view View original

311–320 of 500 posts

Re: Shell script best practices, from a decade of scripting things

#311

Post by ducktective »

maximus-decimus wrote: post #288
ducktective wrote: post #190

Earlier quoted context omitted.

>However, if you instead organize all your data in a format that's sympathetic to line-oriented processing on stdin-stdout, then shell will work with you instead of against. Not even that is necessary. Just use structured data formats like json. If you are consuming some API that is not json but still structured, use `rq` to convert it to json. Then use `jq` to slice and dice through the data. dmenu + fzf + jq + curl…

How do you use dmenu for your shell script? to launch it? to prompt the user for input while it's running? Do you have an example of a script you wrote?

Yes, for creating ad-hoc mini-UIs so the user can select an option. Same with fzf, but it's terminal-bound (rather than X-bound).

The scripts are similar to this one:

https://github.com/debxp/dmenu-scripts/blob/master/dmenu-kil...

Re: Shell script best practices, from a decade of scripting things

#312

Post by usrbinbash »

lbhdc wrote: post #287
usrbinbash wrote: post #264

Earlier quoted context omitted.

> What is the Shell paradigm? I would argue that it's line-oriented pipelines. Which python can do realitively well, by using the `subprocess` module. Here is an example including a https://porkmail.org/era/unix/award (useless use of cat) finding all title lines in README.md and uppercasing them with `tr` import subprocess as sp cat = sp.Popen( ["cat", "README.md"], stdout=sp.PIPE, ) grep = sp.Popen( ["grep", "#"], s…

> But on the other side of that coin its alot easier in python to do a complex regular expression I am not sure I would agree. Sed fills this role quite nicely. cat README.md | grep # | tr '[:lower:] [:upper:]' | sed 's/something/something_else/'

Now do that again, but this time the regular expression is controlled by 2 command line params, one which gives it the substitution, the other one is a boolean switch that tells it whether to ignore case. And the script has to give a good error if the substitution isn't a valid regular expression. It should also give me a helptext for its command line options if I ask it with `-h, --h`.

In python I can use `opt/argparse`, and use the error output from `re.compile` to do this.

Of course this is also possible in bash, but how easy is it to code in comparison, and how maintainable is the result?

Re: Shell script best practices, from a decade of scripting things

#313

Post by irrational »

snidane wrote: post #204
xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

Shell and SQL make you 10x productive over any alternative. Nothing even comes close. I've seen people scrambling for 1 hours to write some data munging, then spend another 1 hour to run it through a thread pool to utilize those cores , while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes. What Python is to Java, Shell is to Python. It sp…

> while somebody comfortable is shell writes a parallelized one liner

Do you have an example of this? I didn’t even know you could make sql calls in scripts.

Re: Shell script best practices, from a decade of scripting things

#314

Post by Beltalowda »

monknomo wrote: post #293
Beltalowda wrote: post #51

Earlier quoted context omitted.

I personally really dislike fish as an interactive shell as it's just so busy. Things keep popping up, everything is in so many different colours, etc. It's great if you like that sort of stuff, but I really appreciate a "quiet" environment. This is also why I use Vim: all the IDEs I tried are just so "busy". I was only talking about scripting; I know fish scripting is different, but I have no idea if it's any good.…

if you want `fish_config` opens up an easy editor for changing all the colors to whatever you find quiet and soothing. You have a level of control over things popping up too

Quite a few things can't be disabled; for example AFAIK it doesn't offer a way to disable the autocomplete altogether, or the "fuzzy" matching. I really dislike these things. Fish is a great shell, but very opinionated which is great if your preferences align with that, and not-so-great if they don't. Which is fine because it makes the project better for those who do want these things, and not every project needs to cater to everyone.

Re: Shell script best practices, from a decade of scripting things

#315

Post by bravetraveler »

foobarian wrote: post #275
ketanmaheshwari wrote: post #234

Earlier quoted context omitted.

Awk is awesome but saying it literally takes 1 hour to properly learn it is a bit overselling.

All you need to do is learn that cmd | awk '{ $5 }' will print out the 5th word as delimited by one or more whitespace characters. Regexes support this easily but are cumbersome to write on the command line.

Doing that, maybe with some inline concatenation to make a new structure, and this are about all I use:

Printing based on another field, example gets UIDs >= 1000:

    awk -F: '$3 >= 1000 {print $0}' /etc/passwd
It can do plenty of magic, but knowing how to pull fields, concat them together, and select based on them cover like 99% of the things I hope to do with it

Re: Shell script best practices, from a decade of scripting things

#316

Post by justanotherbody »

irrational wrote: post #313
snidane wrote: post #204

Earlier quoted context omitted.

Shell and SQL make you 10x productive over any alternative. Nothing even comes close. I've seen people scrambling for 1 hours to write some data munging, then spend another 1 hour to run it through a thread pool to utilize those cores , while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes. What Python is to Java, Shell is to Python. It sp…

> while somebody comfortable is shell writes a parallelized one liner Do you have an example of this? I didn’t even know you could make sql calls in scripts.

mysql, psql etc. let you issue sql from the command line

I don't do much sql in bash scripts but I do keep some wrapper scripts that let me run queries from stdin to databases in my environment

Re: Shell script best practices, from a decade of scripting things

#317

Post by oftenwrong »

IYasha wrote: post #84

Mostly agree, but I add more. 1. end all your lines C-style; this may save your life many times; 2. declare -is variables and -r CONSTANTS at the beginning, again, C-style; 3. print TIMESTAMP="$(date +%Y-%m-%d\ %H:%M:%S)"; where appropriate if your script logs its job; 4. Contrary to OPs reommendation I strongly try to stick to pure SH compatibility in smaller acripts so they can run on routers, TVs, androids and oth…

I like to use:

    date -u +%Y-%m-%dT%TZ
because the time zone is unambiguous, the command works with POSIX date, and it's valid under both ISO 8601 and RFC 3339.

Re: Shell script best practices, from a decade of scripting things

#318

Post by [deleted] »

ketanmaheshwari wrote: post #234
j0hnyl wrote: post #206

Earlier quoted context omitted.

I noticed that I became so much more quick after taking 1 hour to properly learn awk. Yes, it literally takes about 1 hour.

Awk is awesome but saying it literally takes 1 hour to properly learn it is a bit overselling.

[deleted]

Re: Shell script best practices, from a decade of scripting things

#319

Post by metadat »

ketanmaheshwari wrote: post #234
j0hnyl wrote: post #206

Earlier quoted context omitted.

I noticed that I became so much more quick after taking 1 hour to properly learn awk. Yes, it literally takes about 1 hour.

Awk is awesome but saying it literally takes 1 hour to properly learn it is a bit overselling.

Awk is an amazingly effective tool for getting things done quickly.

Submitted yesterday:

Learn to use Awk with hundreds of examples

https://github.com/learnbyexample/Command-line-text-processi...

https://news.ycombinator.com/item?id=33349930

Re: Shell script best practices, from a decade of scripting things

#320

Post by 2devnull »

j0hnyl wrote: post #243
ketanmaheshwari wrote: post #234

Earlier quoted context omitted.

Awk is awesome but saying it literally takes 1 hour to properly learn it is a bit overselling.

I really don't think so! If you have experience with any scripting, you can fully grok the fundamentals of awk in 1 hour. You might not memorize all the nuances, but you can establish the fundamentals to a degree that most things you would try to achieve would take just a few minutes of brushing up. For those that haven't taken the time yet, I think this is a good place to start: https://learnxinyminutes.com/docs/awk…

I feel like I do this every three years then proceed to never use it. Then I read a post on hn and think about how great it could be; rinse and repeat
Post reply on HN