Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bash patterns I use weekly

will-keleher.com

Post reply on HN
Print view View original

61–70 of 115 posts

Re: Bash patterns I use weekly

#61

Post by Arch-TK »

This post and comments section means I no longer wonder why 99% of shell scripts I come across look inept. I'm sorry guys but seriously please actually learn bash (and ideally not from this blog post). There's so many things wrong in the post and the comments that it's difficult to enumerate.

To start with, if you ever feel the need to write a O(n) for loop for finding which commit broke your build, you DID need git-bisect.

Definitely DONT'T wait for PIDs like that and if you do want to write code like that, maybe actually use the arrays bash provides?

If your complaint is that for a in b c d; do ...; done is unclear, then maybe also use lists there, because what's definitely LESS clear is putting things in a variable and relying on splitting.

And most importantly, DO quote things (except when it's unnecessary).

Re: Bash patterns I use weekly

#62

Post by edgyquant »

jolmg wrote: post #40
edgyquant wrote: post #38

Earlier quoted context omitted.

It’s a trade off. I had to use a piped loop earlier this year to extract fallout new Vegas mods and textures since they all had spaces in their file names. For this it was perfect to pipe a list of the names to a loop, but for 99% of things I just use a for loop

Yup. Nearly everything has tradeoffs. BTW, the problem I mentioned earlier can be avoided by using ` $ x=1 $ seq 5 | while read n; do (( x++ )); done $ echo $x 1 $ while read n; do (( x++ )); done Almost makes me wonder what the benefit of preferring a pipe here is. I guess it's just about not having to specify what part of the pipeline is in the same shell.

It’s funny I’ve been using Linux for a decade and a half, professionally for about half that time, and yet I still go to python when arithmetic is involved. I’ve been learning a lot about the shell lately it’s like I did the bare minimum with bash just to be able to run programs and slightly automate things and it took this long for it to click with me that it’s a productive programming language in its own right (and probably faster than python.)

Re: Bash patterns I use weekly

#63

Post by edgyquant »

jolmg wrote: post #43
edgyquant wrote: post #37

Earlier quoted context omitted.

For me I always used for loops and only recently (after a decade of using Linux daily) have learned about the power of piped-loops. It’s strange to me you are more comfortable with those than for loops, but I think it does make sense as you’re letting a program generate the list to iterate over. A pain point in for loops is getting that right, e.g. there isn’t a good way to iterate over files with spaces in them usin…

> A pain point in for loops is getting that right, e.g. there isn’t a good way to iterate over files with spaces in them using a for loop If those files came as arguments, you can use a for-loop as long as they're kept in an array: for f in "${files[@]}"; That handles even newlines in the filenames, while I'm not sure if you can handle that with a while-read-loop. IFS=$'\0' doesn't seem to cut it. for-loops seem pref…

My problem was that I had a directory with probably 200+ subdirectories each one, and the files and subdirectories below them, had a couple of spaces in the name. I typically use

    for f in ‘ls’;
For operations like that but it was obviously built on windows (but I run steam on Ubuntu) and I never interact with windows so tbh I had never thought of this problem before.

Re: Bash patterns I use weekly

#64

Post by chasil »

edgyquant wrote: post #62
jolmg wrote: post #40

Earlier quoted context omitted.

Yup. Nearly everything has tradeoffs. BTW, the problem I mentioned earlier can be avoided by using ` $ x=1 $ seq 5 | while read n; do (( x++ )); done $ echo $x 1 $ while read n; do (( x++ )); done Almost makes me wonder what the benefit of preferring a pipe here is. I guess it's just about not having to specify what part of the pipeline is in the same shell.

It’s funny I’ve been using Linux for a decade and a half, professionally for about half that time, and yet I still go to python when arithmetic is involved. I’ve been learning a lot about the shell lately it’s like I did the bare minimum with bash just to be able to run programs and slightly automate things and it took this long for it to click with me that it’s a productive programming language in its own right (and…

With the original Borne shell, the exper command was used for arithmetic.

The POSIX shell allows the $(( form for native shell arithmetic, but not the (( alternative found in bash and Korn.

Re: Bash patterns I use weekly

#65

Post by chasil »

edgyquant wrote: post #63
jolmg wrote: post #43

Earlier quoted context omitted.

> A pain point in for loops is getting that right, e.g. there isn’t a good way to iterate over files with spaces in them using a for loop If those files came as arguments, you can use a for-loop as long as they're kept in an array: for f in "${files[@]}"; That handles even newlines in the filenames, while I'm not sure if you can handle that with a while-read-loop. IFS=$'\0' doesn't seem to cut it. for-loops seem pref…

My problem was that I had a directory with probably 200+ subdirectories each one, and the files and subdirectories below them, had a couple of spaces in the name. I typically use for f in ‘ls’; For operations like that but it was obviously built on windows (but I run steam on Ubuntu) and I never interact with windows so tbh I had never thought of this problem before.

The GNU way for handling files that have inconvenient characters in their names is:

    find ... -print0 | xargs -0 ...
It makes all the problems go away.

Re: Bash patterns I use weekly

#66

Post by gcmeplz »

Arch-TK wrote: post #61

This post and comments section means I no longer wonder why 99% of shell scripts I come across look inept. I'm sorry guys but seriously please actually learn bash (and ideally not from this blog post). There's so many things wrong in the post and the comments that it's difficult to enumerate. To start with, if you ever feel the need to write a O(n) for loop for finding which commit broke your build, you DID need git-…

> Definitely DONT'T wait for PIDs like that and if you do want to write code like that, maybe actually use the arrays bash provides?

What pattern would you recommend for waiting for PIDs / parallelizing commands & preserving exit codes? Fair point about arrays being a better fit rather than a string there.

Re: Bash patterns I use weekly

#67

Post by l0b0 »

marginalia_nu wrote: post #44
l0b0 wrote: post #31

> 1. Find and replace a pattern in a codebase with capture groups > git grep -l pattern | xargs gsed -ri 's|pat(tern)|\1s are birds|g' Or, in IDEA, Ctrl-Shift-r, put "pat(tern)" in the first box and "$1s are birds" in the second box, Alt-a, boom. Infinitely easier to remember, and no chance of having to deal with any double escaping.

Using an IDE kind of handicaps you to only working with your IDE though. The shell works everywhere for every use case.

I've heard this many times, and I don't really understand the argument. I've used the shell plenty of times for this sort of work, but it's much more complicated to do certain things correctly than in an IDE. Things like looping over any filename, escaping arbitrary strings for use within sed patterns, xargs, multi-line replacements and regex lookahead require quite some in-depth knowledge, trial and error, and sometimes installing special tools because the defaults don't support the relevant patterns (like path NUL terminators or lookaheads). I don't have to deal with any of these in IDEA.

Re: Bash patterns I use weekly

#68

Post by IndexPointer »

gcmeplz wrote: post #66
Arch-TK wrote: post #61

This post and comments section means I no longer wonder why 99% of shell scripts I come across look inept. I'm sorry guys but seriously please actually learn bash (and ideally not from this blog post). There's so many things wrong in the post and the comments that it's difficult to enumerate. To start with, if you ever feel the need to write a O(n) for loop for finding which commit broke your build, you DID need git-…

> Definitely DONT'T wait for PIDs like that and if you do want to write code like that, maybe actually use the arrays bash provides? What pattern would you recommend for waiting for PIDs / parallelizing commands & preserving exit codes? Fair point about arrays being a better fit rather than a string there.

Try GNU parallel.

Re: Bash patterns I use weekly

#69

Post by dang »

Arch-TK wrote: post #61

This post and comments section means I no longer wonder why 99% of shell scripts I come across look inept. I'm sorry guys but seriously please actually learn bash (and ideally not from this blog post). There's so many things wrong in the post and the comments that it's difficult to enumerate. To start with, if you ever feel the need to write a O(n) for loop for finding which commit broke your build, you DID need git-…

Please don't call names or post supercilious putdowns. It's not the culture we want here.

If you know more than others, that's wonderful and it's great to share some of what you know so the rest of us can learn. But please do it without putdowns, and please do it in a way people can actually learn from. "Definitely DONT'T wait for PIDs like that" doesn't actually explain anything.

https://news.ycombinator.com/newsguidelines.html

Re: Bash patterns I use weekly

#70

Post by 1vuio0pswjnm7 »

To conserve host resources RFC 2616 recommends making multiple HTTP requests over a single TCP connection ("HTTP/1.1 pipelining").

The cURL project said it never properly suported HTTP/1.1 pipelining and in 2019 it said it was removed once and for all.

https://daniel.haxx.se/blog/2019/04/06/curl-says-bye-bye-to-...

Anyway, curl is not needed. One can write a small program in their language of choice to generate HTTP/1.1, but even a simple shell script will work. Even more, we get easy control over SNI which curl binary does have have.

There are different and more concise ways, but below is an example, using the IFS technique.

This also shows the use of sed's "P" and "D" commands (credit: Eric Pement's sed one-liners).

Assumes valid, non-malicious URLs, all with same host.

Usage: 1.sh 

       #!/bin/sh
       (IFS=/;while read w x y z;do
       case $w in http:|https:);;*)exit;esac;
       case $x in "");;*)exit;esac;
       echo $y > .host
       printf '%s\r\n' "GET /$z HTTP/1.1";
       printf '%s\r\n' "Host: $y";
       # add more headers here if desired;
       printf 'Connection: keep-alive\r\n\r\n';done|sed 'N;$!P;$!D;$d';
       printf 'Connection: close\r\n\r\n';
       ) >.http
       read x
Post reply on HN