Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Shell script best practices, from a decade of scripting things

sharats.me

Post reply on HN
Print view View original

391–400 of 500 posts

Re: Shell script best practices, from a decade of scripting things

#391

Post by AtlasBarfed »

xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

Hard Disagree. Bash programming:

- no standard unit testing

- how do you debug except with printlns? Fail.

- each line usually takes a minimum of 10 minutes to debug unless you've done bash scripting for... ten years

- basic constructs like the arg array are broken once you have special chars and spaces and want to pass those args to other commands. and UNICODE? Ha.

- standard library is nil, you're dependent on a hodgepodge of possibly installed programs

- there is no dependency resolution or auto-install of those programs or libraries or shell scripts. since it is so dependent on binary programs, that's a good thing, but also sucks for bash programmers

- horrid rules on type conversions, horrid syntax, space-significant rules

- as TFA shows, basic error checking and other conventions is horrid, yeah I want a crap 20 line header for everything

- effective bash is a bag of tricks. Bag of tricks programming is shit. You need to do ANYTHING in it for parsing, etc? Copy paste in functions is basically the solution.

- I'm not going to say interpreter errors are worse than C++ errors, but it's certainly not anything good.

Honestly since even effing JAVA added a hashbang ability, I no longer need bash.

Go ahead, write some bash autocompletion scripts in bash. Lord is that awful. Try writing something with a complex options / argument interface and detect/parse errors in the command line. Awful.

Bash is basically software engineering from the 1970s, oh yeah, except take away the word "engineering". Because the language is actively opposed to anything that "engineering" would entail.

Re: Shell script best practices, from a decade of scripting things

#392

Post by chasil »

j0hnyl wrote: post #243
ketanmaheshwari wrote: post #234

Earlier quoted context omitted.

Awk is awesome but saying it literally takes 1 hour to properly learn it is a bit overselling.

I really don't think so! If you have experience with any scripting, you can fully grok the fundamentals of awk in 1 hour. You might not memorize all the nuances, but you can establish the fundamentals to a degree that most things you would try to achieve would take just a few minutes of brushing up. For those that haven't taken the time yet, I think this is a good place to start: https://learnxinyminutes.com/docs/awk…

Brian Kernighan covers the entire [new] awk language in 40 pages - chapter 2.

There are people who have asked me scripting questions for over a decade, who will not read this for some reason.

It could be read in an hour, but not fully retained.

https://archive.org/download/pdfy-MgN0H1joIoDVoIC7/

Re: Shell script best practices, from a decade of scripting things

#394

Post by pcthrowaway »

ducktective wrote: post #190
xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

>However, if you instead organize all your data in a format that's sympathetic to line-oriented processing on stdin-stdout, then shell will work with you instead of against. Not even that is necessary. Just use structured data formats like json. If you are consuming some API that is not json but still structured, use `rq` to convert it to json. Then use `jq` to slice and dice through the data. dmenu + fzf + jq + curl…

Can you give an example of how you'd use rq in this pipeline? I'm not finding any good examples

Re: Shell script best practices, from a decade of scripting things

#395

Post by chasil »

jb3689 wrote: post #384
snidane wrote: post #204

Earlier quoted context omitted.

Shell and SQL make you 10x productive over any alternative. Nothing even comes close. I've seen people scrambling for 1 hours to write some data munging, then spend another 1 hour to run it through a thread pool to utilize those cores , while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes. What Python is to Java, Shell is to Python. It sp…

I feel like the reasons for this are: * Shell scripts force you to think in a more scalable way (data streams) * Shell scripts compose rich programs rather than simplistic functions * Shells encourage you to program with a rich, extensible feature set (ad-hoc I/O redirection, files) The only times I don’t like shell scripts are when dealing with regex and dealing with parallelism

The POSIX shell does not implement regex.

What is used both in case/esac and globbing are "shell patterns." They are also found in variable pattern removal with ${X% and ${X#.

In "The Unix Programming Environment," Kernighan and Pike apologized for these close concepts that are easily mistaken for one another.

"Regular expressions are specified by giving special meaning to certain characters, just like the asterix, etc., used by the shell. There are a few more metacharacters, and, regrettably, differences in meanings." (page 102)

Bash does implement both patterns and regex, which means discerning their difference becomes even more critical. The POSIX shell is easier in memory for this reason, and others.

http://files.catwell.info/misc/mirror/

Re: Shell script best practices, from a decade of scripting things

#396

Post by partdavid »

Beltalowda wrote: post #66
pizza234 wrote: post #31

Earlier quoted context omitted.

I don't know fish, but I don't consider zsh a step in the right direction, as it tries to be just a cleaned up Bash, which is not enough. There is a general problem in the fact that a radical evolution of glue languages wouldn't be popular because devs rather use Python, and small evolutions wouldn't be popular (ie. zsh), because they end up being confusing (since they're still close to Bash) and not bringing signifi…

zsh is not a "cleaned-up bash"; it's more of a clone of ksh (closed source at the time), with some csh features added in, as well as their own inventions. bash and zsh appeared at roughly the same time, many features were added in zsh first and added to bash later (sometimes much later, and often never). This is kind of a good example of what I meant when people conflate "bash" with "shell". As for your larger point:…

I'm a Unix user and spent almost all of my professional career in bash, and switched to Powershell for my interactive shell a few years ago.

The nice thing with Powershell is that it's not verbose, but the arcane abbreviations are actually quite a bit easier to remember and discover than bash. What mixes people up is that in documented examples and reusable scripts, it makes sense to use the full, canonical name, which looks aesthetically different coming from a Unix background.

Here's what it might actually look like to check a JSON file that has an array of file metadata objects, and delete the ones that have been processed (this includes one user-defined alias, cfj, for "ConvertFrom-Json"):

  gc queue.json | cfj | where status -eq processed | ri
That seems pretty NON-verbose to me, equivalent to how you'd approach this in bash. Do you have jq installed? If you do, perhaps:

  jq '.[] | select(.status == "processed")' queue.yaml | xargs rm
If you don't have jq I think this gets much longer and is either really brittle (you're doing some kind of adhoc parsing that's terrible) or really heavyweight (like pulling in a pure bash JSON library--they exist) or you're using a one-liner from another programming language (Ruby, Python, something). Or maybe you'd complain to whatever was writing 'queue.json' and ask for a friendlier format, like something terrible and one-off you invented because it's easy to "parse" with awk?).

It's even better if you're dealing with network resources:

  $api = https://api/
  irm $api/queue.json | ?{ $_.status -eq processed } | `
    %{ irm -M Delete $api/files/$_.file }
That shows off another alias of Where-Object how you do a foreach loop in a pipeline. And bash:

  api=https://api/
  curl $api/queue.json | \
    jq '.[] | select(.status == "processed") | .file' | \
    while read file; do curl -X DELETE "$api/files/$file"; done
What probably makes you think Powershell is verbose is that, although that's how I type Powershell, it's not how I document it. If I were documenting it for someone else's use, or incorporating that pipeline into a script or module, I'd write it like this:

  Get-Content -Path queue.yaml | ConvertFrom-Json | `
    Where-Object -Property status -Eq 'processed' | Remove-Item
So it's not like it's verbose while you're using it, but verbosity is something you can reach for for clarity when it's desirable. Likewise, you can see the consistent relationship between these commands and their aliases: gc -> Get-Content, cfy -> ConvertFrom-Yaml, ri -> Remove-Item. So you have options for how verbose you need to be. I find it's very useful to have a spelled-out version for commands I don't use all the time, like 'Get-EC2Instance', and consistent verbs so I can make reasonable guesses at the command name for things I'm even less familiar with.

I didn't want to clutter this with too many examples but I'll reiterate that Powershell is a shell. It invokes your Unix commands just fine. For example, if you forgot what the option to Get-Date is, to get a Unix-style timestamp (it's 'get-date -uf %s' so not exactly hard to use): you can just type 'date +%x'. In the example above, I could have used 'cat' and 'xargs rm' instead of 'gc' and 'ri'. So it's not like you have to buy the whole kit and kaboodle right away, either.

Re: Shell script best practices, from a decade of scripting things

#397

Post by KerrAvon »

snidane wrote: post #204
xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

Shell and SQL make you 10x productive over any alternative. Nothing even comes close. I've seen people scrambling for 1 hours to write some data munging, then spend another 1 hour to run it through a thread pool to utilize those cores , while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes. What Python is to Java, Shell is to Python. It sp…

Python is a terrible comparison language here. Of course shell is better than Python for shell stuff; no one should suggest otherwise. Python is extremely verbose, it requires you to be precise with whitespace, and using regex has friction because it's not actually built into the language syntax (unless something has changed very recently).

The comparison should be to perl or Ruby, both of which will fare better than Python for typical shell-type tasks.

Re: Shell script best practices, from a decade of scripting things

#398

Post by partdavid »

einpoklum wrote: post #132
HyperSane wrote: post #113

Earlier quoted context omitted.

sh and bash feel pretty primitive after learning PowerShell.

sh and bash feel a lot more sane and reasonable after learning some PowerShell. PowerShell is mostly not-a-shell.

Can you expand on this? What do you mean by it's "mostly not-a-shell."? I don't understand it.

Re: Shell script best practices, from a decade of scripting things

#399

Post by partdavid »

rockyj wrote: post #25

Speaking of shell, which language do you think has the best interoperatibility with shell commands. I mean, running a command, parsing the output, looping, adding user interaction etc. with the least amount of friction. Ruby used to come close for me, just put the command in backticks `` and write the main logic in Ruby, but I want to hear if there is something better.

Powershell, because it actually is a shell so it's great at easily invoking commands and using their outputs as actual return values, and because it has "programming language" constructs like dependency management in modules, etc.

It has some great tools for user interaction, too, including secure string handling for credentials, a TUI framework, easy parallelism, unit tests and lots more.

Re: Shell script best practices, from a decade of scripting things

#400

Post by wutbrodo »

xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

> I feel like those assertions are more socially-founded opinions than technically-backed arguments

You think the complaints about rickety, unintuitive syntax are "socially founded"? I can't think of another language that has so many pointless syntax issues every time I revisit it. I haven't seen a line of Scheme in over a decade, and I'm still fairly sure I could write a simple if condition with less likelihood of getting it wrong than Bash.

I came at it from the other end, writing complex shell scripts for years because of the intuition that python would be overkill. But there was a moment when I realized how irrational this was: shell languages are enough of a garbage fire that Python was trivially the better choice for my scripts the minute flow control enters the picture.

Post reply on HN