251–260 of 323 posts
awk, grep, sort, and pipe. I'm always amazed at how well thought out, simple, functional, and fast the unix tools are. I still prefer to sift through and validate data using these tools rather than use excel or any full-fledged language. Edit: Also "column" to format your output into a table.
Edit: s/files/fields/
This is, in fact, a Useless Use of Cat [1]. POSIX shells have the
figlet Earlier quoted context omitted.
In fact, I don't like people optimizing shell scripts for performance. I mean, shell scripts are slow by design and if you need something fast, you choose the wrong technology in the first place. Instead, shell script should be optimized for readability and portability and I think it is much easier to understand something like 'read | change >write' than 'change write'. So I like to write pipelines like this: cat foo…
I find something like this: grep '^x' to be very readable, as the flow is still visually apparent based on punctuation.
cat input | grep '^x' | sed 's/foo/bar/g'
sed '/^x/s/foo/bar/g' It's a great learning tool for learning a new programming language as well as the interface between the boundaries of the program are very simple.
For example I wrote this recently https://github.com/djhworld/zipit - I'm fully aware you could probably whip up some awk script to do the same, or chain some existing commands together, or someone else has written the same thing, but I've enjoyed the process of writing it and it's something to throw in the tool box - even if it's just for me!
Earlier quoted context omitted.
Well find, cal, and ls don't take input at all. And vi (well vim) in fact does. If you invoke it with a - for the file name argument, it will read standard input into the buffer. I can't comment on emacs as I don't use it much.
Ah, correct on that point. What about standard output? That is, can vi be used in a pipe?
The ! command sends the selection through external command(s) as STDIN and then replaces the selection with STDOUT from the command(s). For example, grep or sort, but can be any command that works with pipes. Buffer is replaced with output (sorted file for example). Undo with U to go back to original data. Redo with R to go forward to transformed data. Command line history is available to add more commmands or correct when you type ! again.
Edit a file. Select block (Visual mode shift-V) and type ! or use the whole file with gg!G command. Type in the commands you need to run.
Vim also reads stdin if you give - as the filename, like “ls -l | vim -“ so you we can use it at the end of a pipe instead of redirecting to a file.
Like I said, I use it as an interactive debugger to assemble pipelines and see the results.
vi / vim filter commands: http://vimdoc.sourceforge.net/htmldoc/change.html#!
Earlier quoted context omitted.
Of course, if any of your commands prompt for input, you'll be disappointed that's not always as easy as it appears on the surface. Does anyone have a better way to do this kind of thing?
The standard is expect [1]. There are also libraries for many programming languages which perform a similar task, such as pexpect [2]. [1] https://core.tcl.tk/expect/index [2] https://pexpect.readthedocs.io/en/stable/
i.e.
prefer `apt-get install -y` over `yes | apt-get install foo`
Earlier quoted context omitted.
Oh, damn. You're exactly right. OK, to save some of my face, this will work: grep 'foo' ... at least in zsh and probably bash.
I don’t like that at all. That creates a subshell and is also less readable than input | grep foo | sed ...
diff And sometimes programs just refuse to read from stdin but do just fine with an unseekable file on the command line. True, you do have this:
input | recalcitrant_program /dev/stdin
Earlier quoted context omitted.
Allow programs to specify the type of data they can consume and the type of the data they emit. This is how powershell does it (using the dotnet type system).
And the problem is how can you ensure the output type of one program matches the input type of another . A program emits one type, and the other program accepts another. Something will be needed to transform one type into another. Imagine doing that on the command line.
If you want to see what the endgame of this is when taking the reasoning to the maximum, look at visual dataflow languages such as Max/MSP, PureData, Reaktor, LabVIEW... Like always, simple stuff will be simple ( http://write.flossmanuals.net/pure-data/wireless-connections... ) and complicated stuff will be complicated ( https://ni.i.lithium.com/t5/image/serverpage/image-id/96294i... ). No silver bullet guys, sorry.…
> Like always, simple stuff will be simple ( http://write.flossmanuals.net/pure-data/wireless-connections... ) That is not actually simple because the data is flowing across two completely different message passing paradigms . Many users of Max/MSP and Pd don't understand the rules for such dataflow, even though it is deterministic and laid out in the manual IIRC. The "silver bullet" in Max/MSP would be to only use t…
in one hand, this simplifies the semantics (and it's the approach I've been using in my visual language (https://ossia.io)), but in the other it tanks performances if you have large numbers of nodes... I've worked on Max patches with thousands and thousands of objects - if they were all called in a synchronous way as it's the case for the DSP objects you couldn't have as much ; the message-oriented objects are very useful when you want to react to user input for instance because they will not have to execute nearly as often as the DSP objects, especially if you want a low latency.
Earlier quoted context omitted.
In fact, I don't like people optimizing shell scripts for performance. I mean, shell scripts are slow by design and if you need something fast, you choose the wrong technology in the first place. Instead, shell script should be optimized for readability and portability and I think it is much easier to understand something like 'read | change >write' than 'change write'. So I like to write pipelines like this: cat foo…
I agree with the sentiment, but my critique applies so generally that it must be noted: if a command accepts a filename as a parameter, you should absolutely pass it as a parameter rather than `cat` it over stdin. For example, you can write this pipeline as: grep '^x' foo.txt \ | sed 's/a/b/g' \ | awk '{print $2}' \ | wc -l > bar.txt This is by no means scientific, but I've got a LaTeX document open right now. A quic…
That said, I suspect the example would be much faster if you didn't use the pipeline, because a single tool could do it all (I'm leaving in the substitution and column print that are actually unused in the result):
awk '/^x/{gsub("a","b");print $2; count++}END{print NR}' foo.txt