Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bash patterns I use weekly

will-keleher.com

Post reply on HN
Print view View original

91–100 of 115 posts

Re: Bash patterns I use weekly

#91

Post by hnlmorg »

stewartbutler wrote: post #33

I've always had trouble getting `for` loops to work predictably, so my common loop pattern is this: grep -l -r pattern /path/to/files | while read x; do echo $x; done or the like. This uses bash read to split the input line into words, then each word can be accessed in the loop with variable `$x`. Pipe friendly and doesn't use a subshell so no unexpected scoping issues. It also doesn't require futzing around with arr…

> I've always had trouble getting `for` loops to work predictably, so my common loop pattern is this:

for loops were exactly the pain point that lead me to write my own shell > 6 years ago.

I can now iterate through structured data (be it JSON, YAML, CSV, `ps` output, log file entries, or whatever) and each item is pulled intelligently rather than having to conciously consider a tonne of dumb edge cases like "what if my file names have spaces in them"

eg

    » open https://api.github.com/repos/lmorg/murex/issues -> foreach issue { out "$issue[number]: $issue[title]" }
    380: Fail if variable is missing
    379: Backslashes and code comments
    378: Improve testing facility documentation
    377: v2.4 release
    361: Deprecate `swivel-table` and `swivel-datatype`
    360: `sort` converts everything to a string
    340: `append` and `prepend` should `ReadArrayWithType`
Github repo: https://github.com/lmorg/murex

Docs on `foreach`: https://murex.rocks/docs/commands/foreach.html

Re: Bash patterns I use weekly

#92

Post by themk »

1vuio0pswjnm7 wrote: post #85
themk wrote: post #72

Earlier quoted context omitted.

The curl binary will reuse the TCP connection when fed multiple URLs. Infact it can even use HTTP2 and make the requests in parallel over a single TCP connection. Common pattern I use is to construct URLs with a script and use xargs to feed to curl.

For HTTP/1.1 pipelining, not HTTP/2 which not all websites support, the curl binary must be slower because the program tries to do more than just make HTTP from URLs and send the text over TCP. It tries to be "smart" and that slows it down. But dont't take my word for it, test it. For example, compare the retrieval speed of the above to something like sed -n '/^http/s/^/url=/p' URLs.txt|curl -K- --http1.1

I was responding to your original comment, which has since been edited:

> When fed mutiple URLs, the curl binary will open multiple TCP connections, consuming more resources on the host.

Which I felt was a bit of an unfair thing to say.

I have no issue with the rest :)

Re: Bash patterns I use weekly

#93

Post by marginalia_nu »

Something1234 wrote: post #76
marginalia_nu wrote: post #45

Earlier quoted context omitted.

I think what we really need is one regex format. You have POSIX, PCRE, and also various degrees of needing to double-escape the slashes to get past whatever language you're using the regex in. Always adds a large element of guesswork even when you are familiar with regular expressions.

https://regex101.com/

Yeah I know all these dialects, the question is, which is the one I'm supposed to use for (this tool)?

Re: Bash patterns I use weekly

#94

Post by marginalia_nu »

l0b0 wrote: post #67
marginalia_nu wrote: post #44

Earlier quoted context omitted.

Using an IDE kind of handicaps you to only working with your IDE though. The shell works everywhere for every use case.

I've heard this many times, and I don't really understand the argument. I've used the shell plenty of times for this sort of work, but it's much more complicated to do certain things correctly than in an IDE. Things like looping over any filename, escaping arbitrary strings for use within sed patterns, xargs, multi-line replacements and regex lookahead require quite some in-depth knowledge, trial and error, and somet…

Of course it requires knowledge, but the point is that when you have that knowledge, it transfers to other things as well, making you far more capable in many more situations.

Re: Bash patterns I use weekly

#95

Post by Arch-TK »

gcmeplz wrote: post #66
Arch-TK wrote: post #61

This post and comments section means I no longer wonder why 99% of shell scripts I come across look inept. I'm sorry guys but seriously please actually learn bash (and ideally not from this blog post). There's so many things wrong in the post and the comments that it's difficult to enumerate. To start with, if you ever feel the need to write a O(n) for loop for finding which commit broke your build, you DID need git-…

> Definitely DONT'T wait for PIDs like that and if you do want to write code like that, maybe actually use the arrays bash provides? What pattern would you recommend for waiting for PIDs / parallelizing commands & preserving exit codes? Fair point about arrays being a better fit rather than a string there.

So in this case there's a couple notable things:

`wait` can take no parameters this means that if you just ran a bunch of things in the background in your script and want to wait for all of them to finish, you don't need to track the PIDs or a loop, you can just `wait`.

`wait` is a bash builtin (in this case) and as such it has no parameter limit (although I am told that actually there are some weird limits but it's very unlikely you will be able to spawn enough processes at once from a bash script to hit the limits). Given an array `pids` you can just do: `wait "${pids[@]}"`

The only problem with the two above approaches is that in the former case, wait loses the return status and in the second case wait loses all but the status of the last ID you pass it, so the third option is:

    pids=()
    do_thing_1 &
    pids+=("$!")
    do_thing_2 &
    pids+=("$!")
    for pid in "${pids[@]}"; do
        wait "$pid" || status=$?
    done
    exit "${status-0}"
Now you only have the issue left that this will report the LAST failing status.

Re: Bash patterns I use weekly

#96

Post by xorcist »

lambic wrote: post #7
geocrasher wrote: post #4

I want to like this, but the for loop is unnecessarily messy, and not correct. for route in foo bar baz do curl localhost:8080/$route done That's just begging go wonky. Should be stuff="foo bar baz" for route in $stuff; do echo curl localhost:8080/$route done Some might say that it's not absolutely necessary to abstract the array into a variable and that's true, but it sure does make edits a lot easier. And, the orig…

Even more correct would be to use an array: stuff=("foo foo" "bar" "baz") for route in "${stuff[@]}"; do curl localhost:8080/"$route" done

The bash syntax for arrays is obtuse and no one will spot when an error creeps in. It's worth to make an effort to say with space separated strings, as long as you are with bash. More advanced data structures is often an indication that it's worth glacing at something like python.

Should you need to handle spaces, it is often much easier to go with newline separated strings and use "| while read".

This contruction has the added benefit of the data not needing to fit in your environment. This can be a real issue, and is not at all obvious when it happens.

Re: Bash patterns I use weekly

#97

Post by randomopining »

ljm wrote: post #52
OskarS wrote: post #3

Earlier quoted context omitted.

Yes, git bisect is the way to go: in addition to the stuff you mentioned, his method only dives into one parent branch of merge commits. git bisect handles that correctly. A gem of a tool, git bisect.

Bisect also does a binary search so if you're looking for one bad commit amongst many others, you'll find it much more quickly than linearly testing commits, one at a time, until you find a working one.

How does bisect help in a large project? Seems like it would be best to use personal expertise to find it

Re: Bash patterns I use weekly

#98

Post by juki »

hnlmorg wrote: post #91
stewartbutler wrote: post #33

I've always had trouble getting `for` loops to work predictably, so my common loop pattern is this: grep -l -r pattern /path/to/files | while read x; do echo $x; done or the like. This uses bash read to split the input line into words, then each word can be accessed in the loop with variable `$x`. Pipe friendly and doesn't use a subshell so no unexpected scoping issues. It also doesn't require futzing around with arr…

> I've always had trouble getting `for` loops to work predictably, so my common loop pattern is this: for loops were exactly the pain point that lead me to write my own shell > 6 years ago. I can now iterate through structured data (be it JSON, YAML, CSV, `ps` output, log file entries, or whatever) and each item is pulled intelligently rather than having to conciously consider a tonne of dumb edge cases like "what if…

Powershell is also a good option nowadays (although a lot of people on HN seem to dismiss it for various, imo rather superficial, reasons).

  PS> (irm https://api.github.com/repos/lmorg/murex/issues) | % { echo "$($_.number): $($_.title)" }
  380: Fail if variable is missing
  379: Backslashes and code comments
  378: Improve testing facility documentation
  377: v2.4 release
  361: Deprecate `swivel-table` and `swivel-datatype`
  360: `sort` converts everything to a string
  340: `append` and `prepend` should `ReadArrayWithType`
Or just

  PS> (irm https://api.github.com/repos/lmorg/murex/issues) | format-table number, title

  number title
  ------ -----
     380 Fail if variable is missing
     379 Backslashes and code comments
     378 Improve testing facility documentation
     377 v2.4 release
     361 Deprecate `swivel-table` and `swivel-datatype`
     360 `sort` converts everything to a string
     340 `append` and `prepend` should `ReadArrayWithType`
Or even `(irm https://api.github.com/repos/lmorg/murex/issues) | select number, title | out-gridview`, which would open a GUI list (with sorting and filtering), but I think that only works on Windows.

Re: Bash patterns I use weekly

#99

Post by hnlmorg »

juki wrote: post #98
hnlmorg wrote: post #91

Earlier quoted context omitted.

> I've always had trouble getting `for` loops to work predictably, so my common loop pattern is this: for loops were exactly the pain point that lead me to write my own shell > 6 years ago. I can now iterate through structured data (be it JSON, YAML, CSV, `ps` output, log file entries, or whatever) and each item is pulled intelligently rather than having to conciously consider a tonne of dumb edge cases like "what if…

Powershell is also a good option nowadays (although a lot of people on HN seem to dismiss it for various, imo rather superficial, reasons). PS> (irm https://api.github.com/repos/lmorg/murex/issues) | % { echo "$($_.number): $($_.title)" } 380: Fail if variable is missing 379: Backslashes and code comments 378: Improve testing facility documentation 377: v2.4 release 361: Deprecate `swivel-table` and `swivel-datatype`…

The reason I dismissed Powershell was that it doesn't always play nicely with existing POSIX tools, which is very much not a superficial reason :)

Murex aims to give Powershell-style types but still working seamlessly with existing CLI tools. An attempt at the best of both worlds. But I'll let others be the judge of that.

It's also worth noting that Powershell wasn't available for Linux when I first built murex so it wasn't an option even if I wanted it to be.

Re: Bash patterns I use weekly

#100

Post by 1vuio0pswjnm7 »

1vuio0pswjnm7 wrote: post #81
1vuio0pswjnm7 wrote: post #70

To conserve host resources RFC 2616 recommends making multiple HTTP requests over a single TCP connection ("HTTP/1.1 pipelining"). The cURL project said it never properly suported HTTP/1.1 pipelining and in 2019 it said it was removed once and for all. https://daniel.haxx.se/blog/2019/04/06/curl-says-bye-bye-to-... Anyway, curl is not needed. One can write a small program in their language of choice to generate HTTP/…

Heres another way to do it without the subshell, using tr. #!/bin/sh IFS=/;while read w x y z;do v=$(echo x|tr x '\34'); case $w in http:|https:);;*)exit;esac; case $x in "");;*)exit;esac; echo $y > .host printf '%s\r\n' "GET /$z HTTP/1.1"; printf '%s\r\n' "Host: $y"; printf 'Connection: keep-alive'$v$v;done \ |sed '$s/keep-alive/close/'|tr '\34\34' '\r\n' > .http; read x

   case $x in "");;*)exit;esac
is better written as

   test ${#x} = 0||exit
Post reply on HN