Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bash patterns I use weekly

will-keleher.com

Post reply on HN
Print view View original

101–110 of 115 posts

Re: Bash patterns I use weekly

#101

Post by gcmeplz »

Arch-TK wrote: post #95
gcmeplz wrote: post #66

Earlier quoted context omitted.

> Definitely DONT'T wait for PIDs like that and if you do want to write code like that, maybe actually use the arrays bash provides? What pattern would you recommend for waiting for PIDs / parallelizing commands & preserving exit codes? Fair point about arrays being a better fit rather than a string there.

So in this case there's a couple notable things: `wait` can take no parameters this means that if you just ran a bunch of things in the background in your script and want to wait for all of them to finish, you don't need to track the PIDs or a loop, you can just `wait`. `wait` is a bash builtin (in this case) and as such it has no parameter limit (although I am told that actually there are some weird limits but it's…

Thanks for the explanations! Tracking if anything fails is normally important for what I'm doing, so I quite like that array-based solution. I really like that `"$pid" || status=$?`; it's much nicer than `if ! wait $pid; then status=1; fi` I have in there.

Re: Bash patterns I use weekly

#102

Post by matt_kantor »

randomopining wrote: post #97
ljm wrote: post #52

Earlier quoted context omitted.

Bisect also does a binary search so if you're looking for one bad commit amongst many others, you'll find it much more quickly than linearly testing commits, one at a time, until you find a working one.

How does bisect help in a large project? Seems like it would be best to use personal expertise to find it

I've only used bisect a handful of times, but I think it can be most useful in large projects with many contributors.

In smaller projects you can spot a bug and often surmise "I bet this is related to Fred's pull request yesterday that updated the Foo module", but in a larger repository where you don't have all the recent history in your head you might not even know where to start looking. In such cases being able to binary search through history is handy.

Re: Bash patterns I use weekly

#103

Post by 1vuio0pswjnm7 »

themk wrote: post #92
1vuio0pswjnm7 wrote: post #85

Earlier quoted context omitted.

For HTTP/1.1 pipelining, not HTTP/2 which not all websites support, the curl binary must be slower because the program tries to do more than just make HTTP from URLs and send the text over TCP. It tries to be "smart" and that slows it down. But dont't take my word for it, test it. For example, compare the retrieval speed of the above to something like sed -n '/^http/s/^/url=/p' URLs.txt|curl -K- --http1.1

I was responding to your original comment, which has since been edited: > When fed mutiple URLs, the curl binary will open multiple TCP connections, consuming more resources on the host. Which I felt was a bit of an unfair thing to say. I have no issue with the rest :)

Although historically it was true, I agree it was an unfair statement thus I removed it. Thank you for the correction. I will be testing curl a bit more; I rarely ever use it. I could never trust cURL for doing fast HTTP/1.1 pipelining so I am admittedly skeptical. It is a moving target that is constantly changing. The author clearly has a bias toward HTTP/2, and never really focused on HTTP/1.1 pipelining support even though HTTP/1.1 has, IME, worked really well for bulk text retrieval from a single host using a wide variety of smaller simpler programs,^1 not a graphical web browser, for at least fifteen years.^2 HTTP/2 is designed for graphical web pages full of advertising and tracking; introduced as a "standard" by a trillion dollar web advertising company who also controls the majority share web browser. Thankfully, HTTP/1.1 does not have that baggage.

1. Original netcat, djb's tcpclient, socat, etc.

2. It could be used to retrieve lots of small binary files too. See phttpget.

Re: Bash patterns I use weekly

#104

Post by [deleted] »

randomopining wrote: post #97
ljm wrote: post #52

Earlier quoted context omitted.

Bisect also does a binary search so if you're looking for one bad commit amongst many others, you'll find it much more quickly than linearly testing commits, one at a time, until you find a working one.

How does bisect help in a large project? Seems like it would be best to use personal expertise to find it

[deleted]

Re: Bash patterns I use weekly

#105

Post by ljm »

randomopining wrote: post #97
ljm wrote: post #52

Earlier quoted context omitted.

Bisect also does a binary search so if you're looking for one bad commit amongst many others, you'll find it much more quickly than linearly testing commits, one at a time, until you find a working one.

How does bisect help in a large project? Seems like it would be best to use personal expertise to find it

This is a hilariously absurd proposition. The existence of `git bisect` necessarily precludes an issue that personal expertise cannot find without some help.

Personal expertise tells you to use a tool like git bisect to find a problem, so you are more effective in your work. The same as it tells you to use gdb to debug a stack trace. Or do you eyeball the CPU executing instructions in realtime?

An absence of personal expertise will convince you that you are smart enough to do it on your own.

Re: Bash patterns I use weekly

#106

Post by saltcured »

1vuio0pswjnm7 wrote: post #103
themk wrote: post #92

Earlier quoted context omitted.

I was responding to your original comment, which has since been edited: > When fed mutiple URLs, the curl binary will open multiple TCP connections, consuming more resources on the host. Which I felt was a bit of an unfair thing to say. I have no issue with the rest :)

Although historically it was true, I agree it was an unfair statement thus I removed it. Thank you for the correction. I will be testing curl a bit more; I rarely ever use it. I could never trust cURL for doing fast HTTP/1.1 pipelining so I am admittedly skeptical. It is a moving target that is constantly changing. The author clearly has a bias toward HTTP/2, and never really focused on HTTP/1.1 pipelining support ev…

I'm not sure if you realize it but connection reuse and pipelining are not the same thing. Curl does connection reuse with HTTP/1.1 but not pipelining. Connection reuse is what was described up-thread, using a single connection to convey many requests and responses. Pipelining is a step further where the client pushes multiple requests down the connection before receiving responses.

Pipelining is problematic with HTTP/1.1 because servers are allowed to close a connection to signal the end of a response, rather than announcing the Content-Length in the response header. The 1.1 protocol also requires that responses to pipelined requests come in the same order as the original requests. This is awkward and most servers do not bother to parallelize request processing and response writing. Even if the client sends requests back-to-back, the responses will come with delays between them.

With HTTP/1.1 curl (like most clients) will do non-pipelined connection reuse. They push one request, read one response, then push the next request, etc. The network traffic will be small bursts with gaps between them, as each request-response cycle takes at least a round-trip delay. This is still faster than using independent connections, where extra TCP connection setup happens per request, and this is even more significant for HTTPS.

Re: Bash patterns I use weekly

#107

Post by 1vuio0pswjnm7 »

saltcured wrote: post #106
1vuio0pswjnm7 wrote: post #103

Earlier quoted context omitted.

Although historically it was true, I agree it was an unfair statement thus I removed it. Thank you for the correction. I will be testing curl a bit more; I rarely ever use it. I could never trust cURL for doing fast HTTP/1.1 pipelining so I am admittedly skeptical. It is a moving target that is constantly changing. The author clearly has a bias toward HTTP/2, and never really focused on HTTP/1.1 pipelining support ev…

I'm not sure if you realize it but connection reuse and pipelining are not the same thing. Curl does connection reuse with HTTP/1.1 but not pipelining. Connection reuse is what was described up-thread, using a single connection to convey many requests and responses. Pipelining is a step further where the client pushes multiple requests down the connection before receiving responses. Pipelining is problematic with HTT…

I do realise it. I have little interest in curl. It is far too complicated with too many features, IMO. Anyway, libcurl used to support "attempted pipelining". See https://web.archive.org/web/20101015021914if_/http://curl.ha... But the number of people writing programs using CURLMOPT_PIPELINING always seemed small. Thus, I started using netcat instead. No need for libcurl. No need for the complexity.

HTTP/1.1 pipelining was nor is ever "problematic" for me. Thus I cannot relate to statements that try to suggest it is "problematic", especially without providing a single example website.

I am not looking at graphical webpages. I am not trying pull resources from multiple hosts. I am retrieving text from a single host, a continuous stream of text. I do not want parallel processing. I do not want asynchronous. I want synchronous. I want responses in the same order as requests. This allows me to use simple methods for verifying responses and processing them. HTTP/2 is far more complicated. It also has bad ideas like "server push" which is absolutely not what I am looking for.

There are some sites that disable HTTP/1.1 pipelining; they will send a Connection: close header. These are not the majority. There are also some sites where there is a noticeable delay before the first reponse or between responses when using HTTP/1.1 pipelining. That is also a small minority of sites. Most have no noticeable delay. Most are "blazingly fast". "HOB blocking" is not important to me if it is so small that I cannot notice it. If HTTP/1.1 pipelining is "blazingly fast" 98% of the time for me, I am going to use it where I can, which, as it happens, is almost everywhere.

Even with the worst possible delay I have ever experienced, HTTP/1.1 pipelining is still faster than curl/wget/lftp/etc. That is in practice, not theory. People who profess expertise in www technical details today struggle to even agree on what "pipelining" means. For example, see https://en.wikipedia.org/wiki/Talk:HTTP_Pipelining. Trying to explain that socket reuse differs from HTTP/1.1 pipelining is not worth the effort and I am not the one qualified to do it. But I am qualified to state that for text retrieval from single host, HTTP/1.1 pipelining works on most websites and is faster than curl. Is it slower than nghttp2. If yes, how much slower. We know the theory. We also know we cannot believe everything we read. Test it.

Re: Bash patterns I use weekly

#108

Post by 1vuio0pswjnm7 »

1vuio0pswjnm7 wrote: post #107
saltcured wrote: post #106

Earlier quoted context omitted.

I'm not sure if you realize it but connection reuse and pipelining are not the same thing. Curl does connection reuse with HTTP/1.1 but not pipelining. Connection reuse is what was described up-thread, using a single connection to convey many requests and responses. Pipelining is a step further where the client pushes multiple requests down the connection before receiving responses. Pipelining is problematic with HTT…

I do realise it. I have little interest in curl. It is far too complicated with too many features, IMO. Anyway, libcurl used to support "attempted pipelining". See https://web.archive.org/web/20101015021914if_/http://curl.ha... But the number of people writing programs using CURLMOPT_PIPELINING always seemed small. Thus, I started using netcat instead. No need for libcurl. No need for the complexity. HTTP/1.1 pipelin…

One more thing: cURL only supported pipelining GET and HEAD. And on www forums where people try to sound like experts, I have read assertions that POST requests cannot be pipelined. Makes sense in theory, but I know I tried pipelining POST requests before and, surprisingly, it worked on at least one site. Using curl's limited "attempted pipelining", one could never discover this, which is another example of how programs with dozens of features can stil be very inflexible. If anyone doubts this is true, I can try to remember the site that answered pipelined POST requests.

Re: Bash patterns I use weekly

#109

Post by yesenadam »

edgyquant wrote: post #62
jolmg wrote: post #40

Earlier quoted context omitted.

Yup. Nearly everything has tradeoffs. BTW, the problem I mentioned earlier can be avoided by using ` $ x=1 $ seq 5 | while read n; do (( x++ )); done $ echo $x 1 $ while read n; do (( x++ )); done Almost makes me wonder what the benefit of preferring a pipe here is. I guess it's just about not having to specify what part of the pipeline is in the same shell.

It’s funny I’ve been using Linux for a decade and a half, professionally for about half that time, and yet I still go to python when arithmetic is involved. I’ve been learning a lot about the shell lately it’s like I did the bare minimum with bash just to be able to run programs and slightly automate things and it took this long for it to click with me that it’s a productive programming language in its own right (and…

I use "python calc" for quick calculations at the command line:

  pc () { python3 -c "print($*)" ; } 

  $ pc 3.5**2.4
  20.219169193375105
Or "awk calc", which seems much faster: (and ^ and ** both work for powers)

  calc () { awk "BEGIN{print $*}" ; }

  $ calc 3.5**2.4
  20.2192

Re: Bash patterns I use weekly

#110

Post by yesenadam »

marginalia_nu wrote: post #45
marcodiego wrote: post #41

We need a simpler regex format. One that allows easy searching and replacing in source code. Of course, some IDE's already to that pretty well, but I'd like to be able to do it from the command line with a stand alone tool I can easily use in scripts. The simplest thing I know that is able to do that is coccinelle, but even coccinelle is not handy enough.

I think what we really need is one regex format. You have POSIX, PCRE, and also various degrees of needing to double-escape the slashes to get past whatever language you're using the regex in. Always adds a large element of guesswork even when you are familiar with regular expressions.

> I think what we really need is one regex format.

See https://xkcd.com/927/ How standards proliferate

Post reply on HN