Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Shell script best practices, from a decade of scripting things

sharats.me

Post reply on HN
Print view View original

351–360 of 500 posts

Re: Shell script best practices, from a decade of scripting things

#351

Post by chungy »

snidane wrote: post #204
xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

Shell and SQL make you 10x productive over any alternative. Nothing even comes close. I've seen people scrambling for 1 hours to write some data munging, then spend another 1 hour to run it through a thread pool to utilize those cores , while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes. What Python is to Java, Shell is to Python. It sp…

> I started using inline 'python -c' more often than the python repl now as it stores the command in shell history and it is then one fzf search away.

Do you not have a ~/.python_history? The exact same search functions are available on the REPL. Ctrl-R, type your bit, bam.

Re: Shell script best practices, from a decade of scripting things

#352

Post by rascul »

usrbinbash wrote: post #264
xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

> What is the Shell paradigm? I would argue that it's line-oriented pipelines. Which python can do realitively well, by using the `subprocess` module. Here is an example including a https://porkmail.org/era/unix/award (useless use of cat) finding all title lines in README.md and uppercasing them with `tr` import subprocess as sp cat = sp.Popen( ["cat", "README.md"], stdout=sp.PIPE, ) grep = sp.Popen( ["grep", "#"], s…

> But on the other side of that coin its alot easier in python to do a complex regular expression (maybe depending on a command line argument) on one of those, using the result in an HTTP request via the `requests` module, packing the results into a digram rendered in PNG and sending it via email.

Doesn't sound so bad. A quick argument parser, a call out to grep or sed, pipe to curl, then to graphviz I guess (I don't really know much about image generation tools though), then compose the mail with a heredoc and run sendmail. Sounds like 10 to 15 lines for a quick and dirty solution.

Re: Shell script best practices, from a decade of scripting things

#353

Post by scbrg »

usrbinbash wrote: post #312
lbhdc wrote: post #287

Earlier quoted context omitted.

> But on the other side of that coin its alot easier in python to do a complex regular expression I am not sure I would agree. Sed fills this role quite nicely. cat README.md | grep # | tr '[:lower:] [:upper:]' | sed 's/something/something_else/'

Now do that again, but this time the regular expression is controlled by 2 command line params, one which gives it the substitution, the other one is a boolean switch that tells it whether to ignore case. And the script has to give a good error if the substitution isn't a valid regular expression. It should also give me a helptext for its command line options if I ask it with `-h, --h`. In python I can use `opt/argpa…

Man, you chose the wrong username, didn't you? ;-)

Re: Shell script best practices, from a decade of scripting things

#354

Post by Beltalowda »

chubot wrote: post #298
blueflow wrote: post #285

Earlier quoted context omitted.

POSIX shell is the standard, not bash.

That ship has sailed, because busybox ash and dash continually implement some of bash features and semantics, which come from ksh. And OSH implements almost all of bash That is, The posix shell spec is missing a lot of reality. It’s not very actively maintained, unfortunately The canonical example is not having local vars, which basically every shell supports

> The posix shell spec is missing a lot of reality. It’s not very actively maintained, unfortunately

POSIX was primarily intended a descriptive specification, rather than a prescriptive.

That is, it attempted to document and standardize the common behaviour found on many platforms during the Great Unix Wars, rather than say "hey we thought of this great new thing and released a spec, go implement it!", which is more how, say, web standards work. I does/did have some of that, but it was never the main goal.

These days "whatever Linux does" is the de-facto standard, for better or worse, and the need for POSIX is much less.

Re: Shell script best practices, from a decade of scripting things

#355

Post by qwertywert_ »

j0hnyl wrote: post #206
snidane wrote: post #204

Earlier quoted context omitted.

Shell and SQL make you 10x productive over any alternative. Nothing even comes close. I've seen people scrambling for 1 hours to write some data munging, then spend another 1 hour to run it through a thread pool to utilize those cores , while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes. What Python is to Java, Shell is to Python. It sp…

I noticed that I became so much more quick after taking 1 hour to properly learn awk. Yes, it literally takes about 1 hour.

The only thing I use AWK for is getting at columns from output, (possibly processing or conditionally doing something on each) what would be the next big use-case?

Re: Shell script best practices, from a decade of scripting things

#356

Post by rustyminnow »

ducktective wrote: post #190
xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

>However, if you instead organize all your data in a format that's sympathetic to line-oriented processing on stdin-stdout, then shell will work with you instead of against. Not even that is necessary. Just use structured data formats like json. If you are consuming some API that is not json but still structured, use `rq` to convert it to json. Then use `jq` to slice and dice through the data. dmenu + fzf + jq + curl…

Do you have examples of concurrent use-cases that xargs and parallel don't satisfy? I discovered parallel recently and was blown away by how much it improves things. I've only really used it in basic scenarios so far, just wondering where its limitations are.

Re: Shell script best practices, from a decade of scripting things

#358

Post by 5e92cb50239222b »

qwertywert_ wrote: post #355
j0hnyl wrote: post #206

Earlier quoted context omitted.

I noticed that I became so much more quick after taking 1 hour to properly learn awk. Yes, it literally takes about 1 hour.

The only thing I use AWK for is getting at columns from output, (possibly processing or conditionally doing something on each) what would be the next big use-case?

I use it frequently to calculate some basic statistics on log file data.

Here's a nice example of something similar: https://drewdevault.com/dynlib

Re: Shell script best practices, from a decade of scripting things

#359

Post by intrepidhero »

snidane wrote: post #204
xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

Shell and SQL make you 10x productive over any alternative. Nothing even comes close. I've seen people scrambling for 1 hours to write some data munging, then spend another 1 hour to run it through a thread pool to utilize those cores , while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes. What Python is to Java, Shell is to Python. It sp…

If I'm interactively composing something I do very much like pipes and shell commands, but if it's a thing I'm going to be running repeatedly then the improved maintainability of a python script, even if it does a lot of subprocess.run, is preferable to me. "Shuffling clunky objects around" seems more documented and organized than "everything is a bytestring".

But different strokes and all that.

Re: Shell script best practices, from a decade of scripting things

#360

Post by ndsipa_pomu »

Macha wrote: post #349
ndsipa_pomu wrote: post #328

Earlier quoted context omitted.

I tried to write a PowerShell script to recursively scan and find files/folders older than a certain date, but kept hitting problems with the length of the path/filenames. As a complete PowerShell noob, I'm sure I was trying to do it the wrong way, but after a few attempts, I gave up and install cygwin instead.

Your problem here is with win32 not powershell I suspect. Your script would probably have worked with powershell on Linux, but windows you need to use UNC paths to get a 2^16 - ~20 character path limit rather than the 256 character path limit of regular paths. Or there's some registry hacks to remove the limit from regular paths.

You're probably right. It was a few years ago and after trying a couple of alternatives, just went with what I know.
Post reply on HN