Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Learn Awk? (2016)

blog.jpalardy.com

Post reply on HN
Print view View original

71–80 of 246 posts

Re: Why Learn Awk? (2016)

#71

Post by jerf »

ratsmack wrote: post #67
syncsynchalt wrote: post #44

Earlier quoted context omitted.

It does not. Compare: $ echo ' 1 2 3' | cut -f2 1 2 3 $ echo ' 1 2 3' | cut -f2 -d' ' $ echo ' 1 2 3' | awk '{print $2}' 2 "-f [...] Output fields are separated by a single occurrence of the field delimiter character."

Also, the field separator (FS) can be a regular expression. FS = "[0-9]"

IIRC, there is an invocation of cut that basically does what I want, but every time I try, I read the manual page for 3 or 4 minutes, craft a dozen non-functional command lines, then type "awk '{ print $6 }'" and move on.

Re: Why Learn Awk? (2016)

#72

Post by berkes »

Last week I threw out AWK and replaced it with Ruby (Could've been Python, Perl or PHP even).

Because AWK is not suited for CSV. Please prove me wrong!

I had to parse 9million lines. Some of which contain "quoted records", others, same column, are unquoted. Some contain comma's, in the fields, most don't. CSV is like that: more like a guideline than actual sense.

Two hours of googling and hacking later, I gave up and rewrote the importer in Ruby, in under 5 minutes.

Lesson learned: I'll stay clear of AWK, when I know a oneliner of Ruby (or Python) can solve it just as well. Because I know for certain the latter can deal with all the edgecases that will certainly pop up.

Re: Why Learn Awk? (2016)

#73

Post by jerf »

Scarbutt wrote: post #18
jerf wrote: post #9

Because for some bizarre reason, "cut" doesn't ship with any decent column selection logic that is the equivalent of awk's $1, $2, etc., even in 2020. That's like 90% of my use of awk right there. I don't know of any easier equivalent of "awk '{ print $2 }'" for what it does. Posted partially so the Internet Correction Squad can froth at the mouth and set me straight, because I'd love to be showed to be wrong here.

And awk doesn't offers cut's column range selection ;)

Absolutely. Everything comes with costs & benefits. But I'm not sure I've, in my entire 23-year professional programming career, ever encountered a fixed-width text format in the wild. I've used cut even so for places where by coincidence the first couple of columns happened to be the same size, but that's really a hack.

Obviously, other people have different experiences which is why I quality it so. (I only narrowly missed it at the beginning, but I started in webdev, and we never quite took direct feeds from the mainframes.) But I don't think it's too crazy to say UNIX shell, to the extent it can be pinned down at all, works most natively with variable-length line-by-line content.

Re: Why Learn Awk? (2016)

#74

Post by eska »

I gave awk a sincere attempt, but I have to say that it wasn't worth it. As soon as one tries to write anything bigger than a one liner the language shows its warts.. I found myself writing lots of helper routines for things that should be part of the base language/library, e.g. sets with existence check. I also had to work around portability issues, because awk is not awk, unlike this post claims. E.g. matching a part of a line is different among different awks.

And some language decisions are just asinine, e.g. how function parameters and variable scope works, fall through by default although you almost never want that, etc..

But hey, you have socket support! Sounds to me like things have developed in the wrong direction.

And of course no one on your team will know awk.

I found the idea of rule-based programming interesting, but the way it interacts with delimiters and sections (switching rules when encountering certain lines) doesn't work well in practice.

I also found the performance to be very disappoinging when I compared it to C and python equivalents.

Re: Why Learn Awk? (2016)

#75

Post by wnoise »

rb808 wrote: post #16
rectang wrote: post #10

Awk, like shell, is a fraught programming environment, full of silent failure and hidden gotchas. Even for one-liners. I use shell because I have to, not because I like it. I dread maintaining shell scripts which have a bunch of awk and sed in them. The Unix ideal of small single-purpose tools and text processing is separable from these old warhorses.

scripts are the antithesis of modern software development. No Unit testing, No CI, no monitoring, no consistency, often no source control. Appeals to me as a hacker but only my scripts are intelligible - which is a bad sign. :)

Every single ones of those things can be done when using scripts.

Re: Why Learn Awk? (2016)

#76

Post by ende »

berkes wrote: post #72

Last week I threw out AWK and replaced it with Ruby (Could've been Python, Perl or PHP even). Because AWK is not suited for CSV. Please prove me wrong! I had to parse 9million lines. Some of which contain "quoted records", others, same column, are unquoted. Some contain comma's, in the fields, most don't. CSV is like that: more like a guideline than actual sense. Two hours of googling and hacking later, I gave up and…

If you’re doing a lot of csv ops on the cli, you might like https://csvkit.readthedocs.io/en/latest/

(It’s Python, and you can use it as a library as well)

Re: Why Learn Awk? (2016)

#77

Post by mandelbrotwurst »

berkes wrote: post #72

Last week I threw out AWK and replaced it with Ruby (Could've been Python, Perl or PHP even). Because AWK is not suited for CSV. Please prove me wrong! I had to parse 9million lines. Some of which contain "quoted records", others, same column, are unquoted. Some contain comma's, in the fields, most don't. CSV is like that: more like a guideline than actual sense. Two hours of googling and hacking later, I gave up and…

I'm just starting to learn awk, would you mind sharing some of the issues / edge cases that you ran into that you found it didn't handle well? Was it maybe just that you found it tricky to write the regular expressions?

Re: Why Learn Awk? (2016)

#78

Post by jerf »

pizza234 wrote: post #57
jerf wrote: post #9

Because for some bizarre reason, "cut" doesn't ship with any decent column selection logic that is the equivalent of awk's $1, $2, etc., even in 2020. That's like 90% of my use of awk right there. I don't know of any easier equivalent of "awk '{ print $2 }'" for what it does. Posted partially so the Internet Correction Squad can froth at the mouth and set me straight, because I'd love to be showed to be wrong here.

> I don't know of any easier equivalent of "awk '{ print $2 }'" for what it does. I'm not sure if you refer spefically to cut, but Perl has something similar and approximaly terse: > echo 'a b c' | perl -lane 'print $F[1]' Also, Perl can slice arrays, which is something that I really miss in Awk.

That is closest, yes. I'd say clearly a couple more things to remember, but if I can get it into my fingers will be just as fluid. Awk's invocation has its own issues around needing to escape the things the shell thinks it owns, too, not that it's at all unique about that.

Re: Why Learn Awk? (2016)

#79

Post by rakoo »

dhruvkar wrote: post #5
sremani wrote: post #2

Honorary mention of Taco Bell Programming. (fits this genre). http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra... Someone ought to write - Zen and the art of Unix tools usage.

>> suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? I don't know enough about the 'real way' or the 'taco bell way', but interested to know --- is this doable the way Ted describes in the article via xargs and wget?

I would assume a combination of

- sed/awk to extract URLs, one by line

- xargs and wget to download each page from the previous output

Re: Why Learn Awk? (2016)

#80

Post by F-0X »

berkes wrote: post #72

Last week I threw out AWK and replaced it with Ruby (Could've been Python, Perl or PHP even). Because AWK is not suited for CSV. Please prove me wrong! I had to parse 9million lines. Some of which contain "quoted records", others, same column, are unquoted. Some contain comma's, in the fields, most don't. CSV is like that: more like a guideline than actual sense. Two hours of googling and hacking later, I gave up and…

> I had to parse 9million lines.

Awk would chew through that no problem.

> Some of which contain "quoted records", others, same column, are unquoted.

In which case, there is the FPAT variable which can be used to define what a field is. FPAT="\"[^\"]\"|[^,]", which means "stuff between quotes, or things that are not commas", would probably have worked for you. (EDIT: Looks like formatting has gotten hold of my FPAT and I don't know how to stop it... hopefully it is still clear where asterisks should be)

> Some contain comma's, in the fields, most don't. CSV is like that: more like a guideline than actual sense.

Well, I would say that's absolutely false. You can't just put the delimiter wherever you fancy and call it a well-formed file. Quoting exists for the unfortunate cases your data includes the delimiting character (ideally the author would have the sense to use a more suitable character, like a tab).

This is just a retort to prevent your post from dissuading readers from awk, which is a fantastic tool. If you actually sit for half and hour and learn it rather than google to cobble together code that works, it is wonderful. I also don't think it is valid to base your judgement of a tool on what was apparently garbage data.

Post reply on HN