Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Shell script best practices, from a decade of scripting things

sharats.me

Post reply on HN
Print view View original

491–500 of 500 posts

Re: Shell script best practices, from a decade of scripting things

#491

Post by sjellis »

snidane wrote: post #204
xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

Shell and SQL make you 10x productive over any alternative. Nothing even comes close. I've seen people scrambling for 1 hours to write some data munging, then spend another 1 hour to run it through a thread pool to utilize those cores , while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes. What Python is to Java, Shell is to Python. It sp…

I expect nushell to massively change how I work:

https://www.nushell.sh/

It's a shell that is actually built for structured data, taking lessons learned from PowerShell and others.

Re: Shell script best practices, from a decade of scripting things

#492

Post by fluential »

every wrote: post #323
foobarian wrote: post #275

Earlier quoted context omitted.

All you need to do is learn that cmd | awk '{ $5 }' will print out the 5th word as delimited by one or more whitespace characters. Regexes support this easily but are cumbersome to write on the command line.

And don't forget the invisible $0 field in awk...

And $NF

Re: Shell script best practices, from a decade of scripting things

#493

Post by fluential »

partdavid wrote: post #474
gpderetta wrote: post #445

Earlier quoted context omitted.

> The only times I don’t like shell scripts are when dealing with regex and dealing with parallelism Wow, for me parallelism is one of the best features of a unix shell and I find it vastly superior to most other programming languages.

Can you expand on the parallelism features you use and what shell? In bash I've basically given up managing background jobs because identifying and waiting for them properly is super clunky; throttling them is impossible (pool of workers) and so for that kind of thing I've had to use GNU parallel (which is its own abstruse mini-language thing and obviously nothing to do with shell). Ad-hoc but correct parallelism and…

GNU parallel

Re: Shell script best practices, from a decade of scripting things

#494

Post by ekianjo »

foobarian wrote: post #278
ekianjo wrote: post #277

Earlier quoted context omitted.

usually you use specific frameworks for that, not pure Python.

I suppose the Python side is a strawman then - who would do that for a small dataset that fits on a machine? Or have I been using shell for too long :-)

I thought the above comment was about datasets that do not fit on ones machine?

Re: Shell script best practices, from a decade of scripting things

#496

Post by NateEag »

palunon wrote: post #333
NateEag wrote: post #327

Earlier quoted context omitted.

Which works not just to preserve the previous statement from internal inconsistency, but also in regards to the incredibly useful Rule of Three ( https://en.m.wikipedia.org/wiki/Rule_of_three_(computer_prog... ).

> Which works not just to preserve the previous statement from internal inconsistency It doesn't. You now have 4 numbers.

Good point. I'm not sure why I thought what I'd written above worked... shrug

Re: Shell script best practices, from a decade of scripting things

#497

Post by prosaole »

snidane wrote: post #204
xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

Shell and SQL make you 10x productive over any alternative. Nothing even comes close. I've seen people scrambling for 1 hours to write some data munging, then spend another 1 hour to run it through a thread pool to utilize those cores , while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes. What Python is to Java, Shell is to Python. It sp…

> getting the shell quoting hell right

Running `parallel --shellquote --shellquote --shellquote` and pasting in the line you want to quote thrice may alleviate some of the pain.

By no means ideal, though.

Re: Shell script best practices, from a decade of scripting things

#498

Post by GoblinSlayer »

sargstuff wrote: post #466
GoblinSlayer wrote: post #453

Earlier quoted context omitted.

I wrote such a program, that runs other programs for heavy lifting but also parses text which you can't possibly do in bash.

bootloader, systemd, or init ? Parsing text isn't anything fancy. It's just knowing what the marker is for a word/item boundary. For bash, that marker is defined in IFS

A build system for single file programs.

Re: Shell script best practices, from a decade of scripting things

#500

Post by a1a1a »

One of my favorite shell script snippet is prepending timestamp to every output with the help of ts command of moreutils package, meanwhile write to log file at the same time: https://unix.stackexchange.com/questions/26728/prepending-a-...

exec &> >( ts '[%Y-%m-%d.%H:%M:%S] ' | tee ${LOGFILENAME} )

Post reply on HN