Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Learn Awk? (2016)

blog.jpalardy.com

Post reply on HN
Print view View original

81–90 of 246 posts

Re: Why Learn Awk? (2016)

#81

Post by buzer »

jerf wrote: post #73
Scarbutt wrote: post #18

Earlier quoted context omitted.

And awk doesn't offers cut's column range selection ;)

Absolutely. Everything comes with costs & benefits. But I'm not sure I've, in my entire 23-year professional programming career, ever encountered a fixed-width text format in the wild. I've used cut even so for places where by coincidence the first couple of columns happened to be the same size, but that's really a hack. Obviously, other people have different experiences which is why I quality it so. (I only narrowly…

> But I'm not sure I've, in my entire 23-year professional programming career, ever encountered a fixed-width text format in the wild.

SAP comes to mind. I think it does support various different formats, but for reason or another fixed-width seemed to be some kind of default value (that's what I usually got when I asked for SAP feed at least, but that was years ago).

Re: Why Learn Awk? (2016)

#82

Post by dkarl »

jerf wrote: post #9

Because for some bizarre reason, "cut" doesn't ship with any decent column selection logic that is the equivalent of awk's $1, $2, etc., even in 2020. That's like 90% of my use of awk right there. I don't know of any easier equivalent of "awk '{ print $2 }'" for what it does. Posted partially so the Internet Correction Squad can froth at the mouth and set me straight, because I'd love to be showed to be wrong here.

I define aliases c1, c2, c3, c4, etc. in my .bashrc as "awk '{print $1}'" etc.

But it's nice to have awk for the slightly more complicated cases, up until it's easier to use Python or another language.

Re: Why Learn Awk? (2016)

#83

Post by bloopernova »

I use awk because there's an almost 100% chance that it's going to be installed on any unix system I can ssh into.

I use awk because I like to visually refine my output incrementally. By combining awk with multiple other basic unix commands and pipes, I can get the data that I want out of the data I have. I'm not writing unit tests or perfect code, I'm using rough tools to do a quick one-off job.

For instance, "mail server x is getting '81126 delayed delivery' from google messages in the logs, find out who is sending those messages".

# get all the lines with the 81126 message. Get the queue IDs, exclude duplicates, save them in a file.

cat maillog.txt | grep 81126 | awk '{print $6}' | sort | uniq | cut -d':' -f1 > queue-ids.txt

# Grep for entries in that file, get the from addresses, exclude duplicates.

cat maillog.txt | grep -F -f queue-ids.txt | grep 'from=' -f1 | sort | uniq

Each of those 2 one-liners was built up pipe-by-pipe, looking at the output, finding what I needed. It's not pretty, it's not elegant, but it works. I'm sure there's a million ways that a thousand different languages could do this more elegantly, but it's what I know, and it works for me.

Re: Why Learn Awk? (2016)

#84

Post by btilly »

jerf wrote: post #73
Scarbutt wrote: post #18

Earlier quoted context omitted.

And awk doesn't offers cut's column range selection ;)

Absolutely. Everything comes with costs & benefits. But I'm not sure I've, in my entire 23-year professional programming career, ever encountered a fixed-width text format in the wild. I've used cut even so for places where by coincidence the first couple of columns happened to be the same size, but that's really a hack. Obviously, other people have different experiences which is why I quality it so. (I only narrowly…

I can confirm that they are not a common problem.

Admittedly I have encountered fixed-width text formats in the wild. But the last such occasion was about 15 years ago. (It was for interacting with a credit card processor to issue reward cards.)

Re: Why Learn Awk? (2016)

#85

Post by Eikon »

umvi wrote: post #23
rb808 wrote: post #16

Earlier quoted context omitted.

scripts are the antithesis of modern software development. No Unit testing, No CI, no monitoring, no consistency, often no source control. Appeals to me as a hacker but only my scripts are intelligible - which is a bad sign. :)

I'm not actually convinced unit testing is all that valuable, unless the unit under test has a very clear input -> output transformation (like algorithms, string utilities, etc). If it doesn't (and most units don't), unit tests just encumber you.

I’m really glad to read that.

I never understood the whole religion around unit tests. Integration tests are often far easier to write and far more valuable.

Like you said, unit tests are really nice when testing for a known, expected output.

Unit tests that are effectively testing mocks and crazy stubs because your method has side effects? Not for me.

Re: Why Learn Awk? (2016)

#86

Post by masklinn »

jerf wrote: post #71
ratsmack wrote: post #67

Earlier quoted context omitted.

Also, the field separator (FS) can be a regular expression. FS = "[0-9]"

IIRC, there is an invocation of cut that basically does what I want, but every time I try, I read the manual page for 3 or 4 minutes, craft a dozen non-functional command lines, then type "awk '{ print $6 }'" and move on.

> IIRC, there is an invocation of cut that basically does what I want

I don't think there is, because cut separates fields strictly on one instance of the delimiter. Which sometimes works out, but usually doesn't.

Most of the time, you have to process the input through sed or tr in order to make it suitable for cut.

The most frustrating and asinine part of cut is its behaviour when it has a single field: it keeps printing the input as-is instead of just going off and selecting nothing, or printing a warning, or anything which would bloody well hint let alone tell you something's wrong.

Just try it: `ls -l | cut -f 1` and `ls -l | cut -f 13,25-67` show exactly the same thing, which is `ls -l`.

cut is a personal hell of mine, every time I try to use it I waste my time and end up frustrated. And now I'm realising that really cut is the one utility which should get rewritten with a working UI. exa and fd and their friends are cool, but I'd guess none of them has wasted as much time as cut.

Re: Why Learn Awk? (2016)

#87

Post by 0xff00ffee »

pizza234 wrote: post #57
jerf wrote: post #9

Because for some bizarre reason, "cut" doesn't ship with any decent column selection logic that is the equivalent of awk's $1, $2, etc., even in 2020. That's like 90% of my use of awk right there. I don't know of any easier equivalent of "awk '{ print $2 }'" for what it does. Posted partially so the Internet Correction Squad can froth at the mouth and set me straight, because I'd love to be showed to be wrong here.

> I don't know of any easier equivalent of "awk '{ print $2 }'" for what it does. I'm not sure if you refer spefically to cut, but Perl has something similar and approximaly terse: > echo 'a b c' | perl -lane 'print $F[1]' Also, Perl can slice arrays, which is something that I really miss in Awk.

PERL is bloatware by comparison and less likely to be installed on distros than AWK. (e.g, embedded or slim distros. that's why you rarely see nonstandard /bin execs in shell scripts).

Re: Why Learn Awk? (2016)

#88

Post by 0xff00ffee »

eska wrote: post #74

I gave awk a sincere attempt, but I have to say that it wasn't worth it. As soon as one tries to write anything bigger than a one liner the language shows its warts.. I found myself writing lots of helper routines for things that should be part of the base language/library, e.g. sets with existence check. I also had to work around portability issues, because awk is not awk, unlike this post claims. E.g. matching a pa…

You realize AWK is about 1/100th the size of Python, right? That's like comparing a Leatherman multi-tool to a Craftsman 2000 piece tool set that weighs 1,000 lbs. This matters significantly when addressing compatibility and when building distros that are space constrained.

Awk is there for a reason: to be small. That's why the O'Reilly press book is called "Sed & Awk", because they were orignally written to work together in the early days of unix dating back to the late 70's. Sed (1974) & Awk (1977) are in the DNA of unix, Python is something totally different.

Re: Why Learn Awk? (2016)

#89

Post by Scarbutt »

shpeedy wrote: post #49
rhombocombus wrote: post #14

It is fast, robust, and frequently far more performant than a lot of modern tools that can be overkill for most data manipulation. I use it all the time in our ETL processes and it always works as advertised.

Perl is much faster[0], with much more features, with bunch of ready to use libraries, with package manager (CPAN), and similar syntax to awk. Why you use awk? [0]: http://rc3.org/2014/08/28/surprisingly-perl-outperforms-sed-...

Because you can learn awk in 1/10 of a time it takes to learn perl.

Re: Why Learn Awk? (2016)

#90

Post by 0xff00ffee »

rectang wrote: post #28
rhombocombus wrote: post #20

Earlier quoted context omitted.

While I agree that the silent failures and "opaqueness" can be off-putting, once you understand the tool and how to implement it in your workflow it is wonderfully efficient. Aside from being syntactically terse I haven't found any compelling reasons not to use it.

I've achieved a high level of expertise in Perl. Even though I can, I won't write complex Perl one liners; instead, I will write tested, documented utilities which someone who has not achieved such a level of mastery has a shot at grokking, maintaining and debugging. Even better if I can write such utilities in a more modern programming language like Go or Rust, if the organization has personnel with expertise in suc…

rectang: "Awk, like shell, is a fraught programming environment, full of silent failure and hidden gotchas. Even for one-liners..."

Also rectang: "Even better if I can write such utilities in a more modern programming language like Go or Rust..."

Thank you for demonstrating how Linux decays due to lack of experience and NIH syndrome.

Post reply on HN