41–50 of 108 posts
I like slicing and dicing with awk, grep and friends too. One thing I find odd that you have to drop to a full language (awk, perl etc) to sum a column of numbers. Am I missing a utility? echo "1\n2\n3\n" | sum # should print 6 with hyphothetical sum command I suppose more generally you could have a 'fold initial op' and: echo "1\n2\n3\n4\n" | fold 0 + # should print 10 echo "1\n2\n3\n4\n" | fold 1 \* # should print…
echo "1\n2\n3\n" | tr '\n' + | bchttp://www.slideshare.net/strands/strands-presentation-at-re...
Earlier quoted context omitted.
Could you expand on why export LC_ALL=C would "make all your commands 3x faster"?
Gnu grep is or was very slow with the UTF-8 locale. Not sure about other commands, perhaps anything that processes text, awk and sed maybe?
Starts off with unnecessary use of cat, e.g., cat file | awk 'cmds'. One can simply do awk 'cmds' file.
I know purists always complain about unnecessary cats, but I always find it useful to start with "head" or "tail" in the first position to figure out my pipeline, and then replace it with cat when it's all working. And if the extra cat is actually making a measurable difference, maybe that's a good signal that it's time to rewrite it in C.
$ cat data.txt | awk '{ print $2+$4,$0 }'|sort|sed '/^0/d'
$ "Don't pipe a cat".
My test doesn't show a speed improvement, but there are less processes running, and less memory consumed.
bch:~ bch$ jot 999999999 2 99 > data.dat
bch:~ bch$ time cat data.dat | awk '{sum +=$1} END {printf "sum: %d\n", sum}'
sum: 50499999412
real 6m21.111s
user 6m15.506s
sys 0m5.711s
PID COMMAND %CPU TIME #TH #WQ #PORTS #MREGS RPRVT RSHRD RSIZE VPRVT VSIZE PGRP PPID STATE UID FAULTS COW MSGSENT MSGRECV
22342 awk 100.7 05:11.84 1/1 0 17 21 52K 212K 340K 17M 2378M 22341 22306 running 501 311 49 73 36
22341 cat 1.1 00:03.94 1/1 0 17 21 272K 212K 548K 17M 2378M 22341 22306 running 501 268 51 73 36
bch:~ bch$ time awk '{sum +=$1} END {printf "sum: %d\n", sum}' ./data.dat
sum: 50499999412
real 6m24.023s
user 6m13.828s
sys 0m2.774s
PID COMMAND %CPU TIME #TH #WQ #PORTS #MREGS RPRVT RSHRD RSIZE VPRVT VSIZE PGRP PPID STATE UID FAULTS COW MSGSENT MSGRECV
22373 awk 100.0 00:30.16 1/1 0 17 21 276K 212K 624K 17M 2378M 22373 22306 running 501 256 46 73 36> cat data.csv | awk -F "|" '{ sum += $4 } END { printf "%.2f\n", sum }' "Don't pipe a cat". My test doesn't show a speed improvement, but there are less processes running, and less memory consumed. bch:~ bch$ jot 999999999 2 99 > data.dat bch:~ bch$ time cat data.dat | awk '{sum +=$1} END {printf "sum: %d\n", sum}' sum: 50499999412 real 6m21.111s user 6m15.506s sys 0m5.711s PID COMMAND %CPU TIME #TH #WQ #PORTS #MREG…
#!/bin/sh
printf %s\\n "$@" | awk -F'\t' '
FILE == "/dev/stdin" {
needle[$0] = 1
next
}
needle[$4] {
print $NF
}
' /dev/stdin /etc/pki/CA/index.txt
Earlier quoted context omitted.
>I recommend "The AWK Programming Language" by Aho, Kernighan, and Weinberger I concur with this recommendation. "The AWK Programming Language", at little over 100 pages, is a classic of programming language instruction. The book jumps right into use cases, it does not waste one's time. This book should be required reading for anyone contemplating writing a handbook on any programming language; my CS bookshelf would…
Sadly it seems very expensive now, $95 on Amazon...