Earlier quoted context omitted.
That's the kind of video I might have downloaded. At least I hope so. Gonna check my backups. update 1 : found it, time to upload.
Thank you kind sir. Waiting for a link
The Mighty Named Pipe
71–80 of 99 posts
Re: The Mighty Named Pipe
#72If you like pipes, then you will love lazy evaluation. It is unfortunate, though, that Unix doesn't support that (operations can block when "writing" only, not when "nobody is reading").
I don't know, I love pipes and I'm on the fence regarding strict vs lazy. BTW: when is nobody reading in pipes? There's always implicit &> stdout added. EDIT: oh, right, named pipes.
If you write to a named pipe, the call to write(2) will block until somebody opens it for reading and begins to read(2) it.
Re: The Mighty Named Pipe
#73How does the > process substitution differ from simply piping the output with | ? For example (from Wikipedia) tee >(wc -l >&2) bigfile.gz vs tee bigfile.gz
[deleted]
Re: The Mighty Named Pipe
#74Earlier quoted context omitted.
I wish they'd supply that too, but they do seem to have gotten better at interpreting literally when it makes sense in context. I've been learning C# and have found, for example, that searches with the term "C#" return the appropriate resources when in the past I'd have probably seen results for C.
Google handles some constructs with punctuation as atomic tokens as special cases. C# and C++ are examples. A# through G# also return appropriate results, for the musical notes. H# and onward through the alphabet do not. .NET is another example. Google will ignore a prepended dot on most words, but .NET is handled specially as an atomic token. I would bet this is a product of human curation, not of algorithms that ha…
Occasionally it's useful to spell out the names of the characters, both when searching and when writing documentation, blog posts, and SO Q&A. That way, searching for "asp.net less than percent hash" might tell you it's the data-bind operator.
Re: The Mighty Named Pipe
#75Earlier quoted context omitted.
>The user does not care in general how something is performed, just that it is performed correctly and with good performance. This is the crux of the matter. With BASH scripting the user does care how a task is preformed as that task maybe system administration, involve sensitive system components, OR sensitive data. Lazy evaluation is great for binary/cpu level optimization. But passing system administration tasks t…
Well, in any case, the problem can be resolved by adding a kernel-level api function that allows one to wait (block) until results are requested from the other end of the pipe.
Why?
The opposite is already true and has the same effect.
Each stage of the pipeline is executed when it has data to execute. So ultimately the main blocking event is IO (normally the first stage in a pipeline). Every other process is automatically marked as blocked, until its stdin is populated by the output of the former. Once its task is complete it re-checks stdin, and if nothing is present blocks itself.
So the execution of each task is controlled by the process who's data that task needs to operate.
In your system why would you want to block the previous step? This would just interfere with the previous+1 step, and you'd have to populate that message further up the chain. This seems needlessly complicated. As you have to add extra IPC.
Re: The Mighty Named Pipe
#76Nice article. Really easy to follow introduction. I only discovered process substitution a few months ago but it's already become a frequently used tool in my kit. One thing that I find a little annoying about unix commands sometimes is how hard it can be to google for them. ' Unless you know to look for "Process Substitution" it can be hard to find information on these things. And that's once you even know these thi…
#bash on freenode, and http://mywiki.wooledge.org/BashFAQ
Re: The Mighty Named Pipe
#77Anybody know of a way to increase the buffer size of pipes? I've experienced cases where piping a really fast program to a slow one caused them both to go slower as the OS pauses first program writing when pipe buffer is full. This seemed to ruin the caching for the first program and caused them both to be slower even though normally pipes are faster as you're not touching disk.
Both mbuffer and pv by default contain fairly large in-memory buffers for pipe data, and accept parameters for particularly large buffers. http://www.maier-komor.de/mbuffer.html http://www.ivarch.com/programs/pv.shtml
Thanks - hoping that there was a built in solution but a buffer program makes sense
Re: The Mighty Named Pipe
#78Once you discover # avoid temporary files when some program needs two inputs: join -e0 -o0,1.1,2.1 -a1 -a2 -j2 -t$'\t' \
one of my favorites is using diff across output of two programs, diff thinks they are files:
diff -u <(zipinfo archive.zip.bak) <(zipinfo archive.zip)
Re: The Mighty Named Pipe
#79Nice article. Really easy to follow introduction. I only discovered process substitution a few months ago but it's already become a frequently used tool in my kit. One thing that I find a little annoying about unix commands sometimes is how hard it can be to google for them. ' Unless you know to look for "Process Substitution" it can be hard to find information on these things. And that's once you even know these thi…
books on UNIX and shells
Re: The Mighty Named Pipe
#80Earlier quoted context omitted.
> # gawk doesn't care if it's given a regular file or the output fd of some process: Something wonderful I found out the other day: Bash executes scripts as it parses them, so you can do all kinds of awful things. For starters, bash will have bash execute an infinite script that looks like echo hello echo hello echo hello ... without trying to load the whole thing first. After that, you can move onto having a script…
That's actually one of the things that I really dislike with bash, that it doesn't read the whole script before executing it. I've been bitten by it before, when I write some long-running script, then e.g. write a comment at the top of it as it's running, and then when bash looks for the next command, it's shifted a bit and I get (at best) a syntax error and have to re-run :-(
There are several ways to get Bash to read the whole thing before executing.
My preferred method is to write a main() function, and call main "$@" at the very end of the script.
Another trick, useful for shorter scripts, is to just wrap the body of the script in {}, which causes the script to be a giant compound command that is parsed before any of it is executed; instead of a list of commands that is executed as read.