There used to be a website shared on HN a lot, which was a table of: Task, SED, Awk.
It was really useful but I haven’t seen it for years.
Why Learn Awk? (2016)
121–130 of 246 posts
Re: Why Learn Awk? (2016)
#122Honorary mention of Taco Bell Programming. (fits this genre). http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra... Someone ought to write - Zen and the art of Unix tools usage.
(warning, mandatory HN contrarian comment)
"This is the opposite of a trend of nonsense called DevOps, where system administrators start writing unit tests and other things to help the developers warm up to them - Taco Bell Programming is about developers knowing enough about Ops (and Unix in general) so that they don't overthink things, and arrive at simple, scalable solutions"
It's not possible for developers to know enough about Ops, just as it's not possible for Ops to know enough about development, because they are different jobs. Moreover, devs are doomed to create terrible solutions because of their job, and Ops are doomed to create kludgy hacks for those terrible solutions because of their job. DevOps is just an attempt to get them to talk to each other frequently so that horrible shit doesn't happen as frequently.
Also, the real Taco Bell programming is actually to use only wget, no xargs. It takes a whole lot of basically every option Wget has, and a very reliable machine with a lot of RAM, but you can crawl millions of pages with just that tool. xargs and find make it worse because you don't get the implicit caching features of the one giant threaded process, so you waste tons of time and disk space re-getting the same pages, re-looking up the same hostnames, etc. (And that's Ops knowledge...)
The Zen of Unix is to try to move towards not using the computer at all. One-liners are part of that path, but so is minimizing the one-liner. http://www.catb.org/~esr/writings/unix-koans/ten-thousand.ht...
Re: Why Learn Awk? (2016)
#123Skip learning of sed and awk and jump straight to perl instead. $ perl --help ... -F/pattern/ split() pattern for -a switch (//'s are optional) -l[octal] enable line ending processing, specifies line terminator -a autosplit mode with -n or -p (splits $_ into @F) -n assume "while ( ) { ... }" loop around program -p assume loop like -n but print line also, like sed -e program one line of program (several -e's allowed,…
Also, because there's a whole culture of one-liners in Perl, you can also conveniently import libraries and call them:
Even though there's frequently value in adding whitespace to programs, many of them are just fine as one liners :)
e.g. this gets the title of a webpage for you: ``` $ perl -Mojo -E 'say g("mojolicious.org")->dom->at("title")->text' ```
Re: Why Learn Awk? (2016)
#124Because for some bizarre reason, "cut" doesn't ship with any decent column selection logic that is the equivalent of awk's $1, $2, etc., even in 2020. That's like 90% of my use of awk right there. I don't know of any easier equivalent of "awk '{ print $2 }'" for what it does. Posted partially so the Internet Correction Squad can froth at the mouth and set me straight, because I'd love to be showed to be wrong here.
I define aliases c1, c2, c3, c4, etc. in my .bashrc as "awk '{print $1}'" etc. But it's nice to have awk for the slightly more complicated cases, up until it's easier to use Python or another language.
awk '{print $1}' can also be written as awk '$0=$1'
Re: Why Learn Awk? (2016)
#125I use awk because there's an almost 100% chance that it's going to be installed on any unix system I can ssh into. I use awk because I like to visually refine my output incrementally. By combining awk with multiple other basic unix commands and pipes, I can get the data that I want out of the data I have. I'm not writing unit tests or perfect code, I'm using rough tools to do a quick one-off job. For instance, "mail…
I know you’re not asking for awk protips but you can prefix the block with a match condition for processing. ... | grep foo | awk ‘{print $6}’ | ... becomes ... | awk ‘/foo/{print $6}’ | ... If you start working this into your awk habits you’ll find delightful little edge cases that you can handle with other expressions before the block (you can, for example, match specific fields).
I always forget about that, and I should try more to remember it. Thank you for the tip!
Re: Why Learn Awk? (2016)
#126Awk is a command I turn to time and again. For me it's the single most valuable command for enabling the piped single-purpose pattern. As an example, if I want a sorted list of all open files under the home directories on CentOs I can do this: lsof | awk '{ print $10 }' | grep ^/home/ | sort | uniq
you can save one command by using sort -u which eliminates dupes in the sort, removing the need for uniq
Tnx, that one I didn't know...
You can drop another command as such:
lsof |awk '($10 ~ /^\/home\//) {print $10}' |sort -uRe: Why Learn Awk? (2016)
#127Earlier quoted context omitted.
> Aside from being syntactically terse I haven't found any compelling reasons not to use it. For me, that's THE reason to use it. It's terseness is what allows it to be efficient enough to be used primarily interactively.
I wholeheartedly agree, but I think that is why a lot of folks get turned off from it.
There is a school of thought in CS that equates terseness to bad programming practices. That is unfortunate. It is possible for software to be terse and well designed, and of course verbose solutions can easily morph into an unmaintainable nightmare.
Re: Why Learn Awk? (2016)
#128It is fast, robust, and frequently far more performant than a lot of modern tools that can be overkill for most data manipulation. I use it all the time in our ETL processes and it always works as advertised.
If you don't need gnu extensions, I've found mawk to be 4x faster than gawk on some scripts
Re: Why Learn Awk? (2016)
#129Earlier quoted context omitted.
I'm not actually convinced unit testing is all that valuable, unless the unit under test has a very clear input -> output transformation (like algorithms, string utilities, etc). If it doesn't (and most units don't), unit tests just encumber you.
I’m really glad to read that. I never understood the whole religion around unit tests. Integration tests are often far easier to write and far more valuable. Like you said, unit tests are really nice when testing for a known, expected output. Unit tests that are effectively testing mocks and crazy stubs because your method has side effects? Not for me.
It is easy to explain: unit tests give a documented proof that you care about code robustness. It is used more for social and psychological effect than for its advantages. In fact, outside a few domains, unit tests make it harder to evolve software because the more tests you write, the harder it is to make changes that move the design in a different direction. This is, by the way, my main problem with verbose techniques of programming: the more you have to code, the harder to make needed changes.
Re: Why Learn Awk? (2016)
#130I gave awk a sincere attempt, but I have to say that it wasn't worth it. As soon as one tries to write anything bigger than a one liner the language shows its warts.. I found myself writing lots of helper routines for things that should be part of the base language/library, e.g. sets with existence check. I also had to work around portability issues, because awk is not awk, unlike this post claims. E.g. matching a pa…
Most complaints that awk doesn't have have this or that feature ignore the fact that awk is not supposed to be used in isolation. Any substantial use of awk has to be tied to other UNIX utilities. I don't think you can, or should, write a medium to large size script completely in awk, the whole idea is to compose it with one or more UNIX commands.