Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing Awk Is Tricky

raygard.net

Post reply on HN
Print view View original

51–60 of 95 posts

Re: Parsing Awk Is Tricky

#51

Post by shawn_w »

forinti wrote: post #47
scbrg wrote: post #46

Earlier quoted context omitted.

Well. I don't know. Those two programs don't really do the same thing. There's an awful lot of comparisons in the second one. After making the awk program more similar to the Perl program, and using mawk instead of gawk (which is quite a bit slower) the numbers look a bit different: $ seq 100000000 > /tmp/numbers $ time perl -MData::Dumper -ne '$n{length($_)}++; END {print Dumper(%n)}' /tmp/numbers $VAR1 = '7'; $VAR2…

I used them both to find the longest line in a file. The Perl option just spits out the number of times each line length occurs. It will get messy if you have many different line lengths (which was not my case). You also have to take into account that awk does not count the line terminator. Let's try the opposite: make the Perl script more like the AWK one. $ time perl -ne 'if(length($_)>$n) {$n=length($_)}; END {pri…

`perl -lne ...` to have perl strip the trailing newlines like awk does. Should give the same result with it.

Re: Parsing Awk Is Tricky

#52

Post by shawn_w »

Chris2048 wrote: post #35
mmsc wrote: post #32

Awk is something that I think every programmer and especially every sysadmin should learn. 8 like the comparison at the end and have never heard of nnawk or bbawk before. I recently made a dashboard to compare four versions of awk output together, since not all awk scripts I'll run the same on each version: https://megamansec.github.io/awk-compare/ I'll have to add those:)

> every programmer and especially every sysadmin should learn There are lots of things "every should learn", usually by people who already did so. I still have a bunch of AI/ML items on that list too. What's the advantage of learning AWK over Perl?

I put off learning awk for literal decades because I knew perl, but then I picked it up and wish I had done so earlier. I still prefer perl for a lot of use cases, but in one-liners, awk's syntax makes working with specific fields a lot more convenient than perl's autosplit mode. `$1` instead of `$F[0]`, basically.

Re: Parsing Awk Is Tricky

#53

Post by LegionMammal978 »

Snoddas wrote: post #44
Chris2048 wrote: post #35

Earlier quoted context omitted.

> every programmer and especially every sysadmin should learn There are lots of things "every should learn", usually by people who already did so. I still have a bunch of AI/ML items on that list too. What's the advantage of learning AWK over Perl?

Both will get you where you want to go, but I don't think the usecase for perl and awk are the same. I reach for awk when my bash-scripts get a bit messy, perl is/was for when I want to build a small application (or nowdays python). But both perl and python require cpan/pip to get the most out of and with awk, I just nead awk.

Is there any particular functionality which does exist in awk, but doesn't exist in Perl or Python without third-party libraries? I've always found "Python + built-in modules" more than sufficient for my text-manipulation needs. (Also, it lets me handle binary data and character data in the same program, which is very useful for certain tasks.)

Re: Parsing Awk Is Tricky

#54

Post by forinti »

shawn_w wrote: post #51
forinti wrote: post #47

Earlier quoted context omitted.

I used them both to find the longest line in a file. The Perl option just spits out the number of times each line length occurs. It will get messy if you have many different line lengths (which was not my case). You also have to take into account that awk does not count the line terminator. Let's try the opposite: make the Perl script more like the AWK one. $ time perl -ne 'if(length($_)>$n) {$n=length($_)}; END {pri…

`perl -lne ...` to have perl strip the trailing newlines like awk does. Should give the same result with it.

You're right. It even makes the times converge.

Re: Parsing Awk Is Tricky

#55

Post by LegionMammal978 »

mananaysiempre wrote: post #22
v3ss0n wrote: post #5

Reading awk as a human is hard too. And performance of awk is crap. A lot slower than most interpreter language out there. I had replaced all the awk scripts in python and everything is a lot faster.

> And performance of awk is crap. [...] I had replaced all the awk scripts in python and everything is a lot faster. My experience points exactly the other way: for data-processing tasks, especially streaming ones, even Gawk is a lot faster than Python (pre-3.11), and apparently I’m not the only one[1]. If you’re not satisfied with Gawk’s performance, though, try Nawk[2] or, even better, Mawk[3]. (And stick to POSIX…

Do you know of any performance comparisons vs. PyPy? I find it works extremely well as a drop-in replacement for CPython when only the built-in modules are needed, which should generally hold for awk-like use cases. Yet some brief searching doesn't seem to yield any numbers.

Re: Parsing Awk Is Tricky

#56

Post by Chris2048 »

rlonstein wrote: post #40
Chris2048 wrote: post #35

Earlier quoted context omitted.

> every programmer and especially every sysadmin should learn There are lots of things "every should learn", usually by people who already did so. I still have a bunch of AI/ML items on that list too. What's the advantage of learning AWK over Perl?

- Awk is defined in POSIX - Awk is on more systems than Perl - Awk has more implementations than Perl

> Awk is defined in POSIX

so?

> Awk is on more systems than Perl

By what metric?

> Awk has more implementations than Perl

so?

Re: Parsing Awk Is Tricky

#57

Post by Chris2048 »

chrsig wrote: post #42
rlonstein wrote: post #40

Earlier quoted context omitted.

- Awk is defined in POSIX - Awk is on more systems than Perl - Awk has more implementations than Perl

awk is also a much smaller language than perl, so it's generally less effort to teach, learn, and read.

Is it not possible to learn a subset of perl?

Re: Parsing Awk Is Tricky

#58

Post by williamcotton »

LegionMammal978 wrote: post #53
Snoddas wrote: post #44

Earlier quoted context omitted.

Both will get you where you want to go, but I don't think the usecase for perl and awk are the same. I reach for awk when my bash-scripts get a bit messy, perl is/was for when I want to build a small application (or nowdays python). But both perl and python require cpan/pip to get the most out of and with awk, I just nead awk.

Is there any particular functionality which does exist in awk, but doesn't exist in Perl or Python without third-party libraries? I've always found "Python + built-in modules" more than sufficient for my text-manipulation needs. (Also, it lets me handle binary data and character data in the same program, which is very useful for certain tasks.)

It’s just that awk has a concise syntax that can make for some really quick one-liners in your terminal prompt. Why spend a minute or two in Python if you can get an answer in 15 seconds instead?

Re: Parsing Awk Is Tricky

#59

Post by hulium »

mmsc wrote: post #32

Awk is something that I think every programmer and especially every sysadmin should learn. 8 like the comparison at the end and have never heard of nnawk or bbawk before. I recently made a dashboard to compare four versions of awk output together, since not all awk scripts I'll run the same on each version: https://megamansec.github.io/awk-compare/ I'll have to add those:)

awk is also not hard to understand, scroll through the Wikipedia page for a few minutes https://en.wikipedia.org/wiki/AWK#Structure_of_AWK_programs

It runs an action for each line in the input (optionally filtered by regex). You get automatic variables $1,$2... for the words in the line split by spaces.

The syntax is almost like a simple subset of Javascript. Builtin functions are similar to C standard library.

If you have input in text that is separated in columns with a delimiter, and you want do simple operations on that (filter, map, aggregate), it can be done quickly with awk.

That's all you need to know about awk.

Re: Parsing Awk Is Tricky

#60

Post by mbivert »

Chris2048 wrote: post #35
mmsc wrote: post #32

Awk is something that I think every programmer and especially every sysadmin should learn. 8 like the comparison at the end and have never heard of nnawk or bbawk before. I recently made a dashboard to compare four versions of awk output together, since not all awk scripts I'll run the same on each version: https://megamansec.github.io/awk-compare/ I'll have to add those:)

> every programmer and especially every sysadmin should learn There are lots of things "every should learn", usually by people who already did so. I still have a bunch of AI/ML items on that list too. What's the advantage of learning AWK over Perl?

> What's the advantage of learning AWK over Perl?

Getting awk in your head (fully) takes about an afternoon: reading the (small and exhaustive) man page, going through a few examples, trying to build a few toys with it. Perl requires much, much more effort.

Great gain/investment ratio.

Post reply on HN