Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Awk in 20 Minutes (2015)

ferd.ca

Post reply on HN
Print view View original

61–70 of 128 posts

Re: Awk in 20 Minutes (2015)

#61

Post by cyrusmg »

liammonahan wrote: post #32

My absolute favorite example of what's possible with awk is this[0] calculator from Ward Cunningham about splitting expenses on a ski trip. It's a really beautiful little piece of code well-adapted to this problem. [0] - https://c2.com/doc/expense/

I am trying to parse the program based on the article above and comments on the link you posted.

> The first occurrence of a variable name defines it as that sum. Subsequent occurrences become the stored value.

With this quote in mind - why is second `SUM` reference replaced with `-138.95` and not `221.81` (stored value) ?

EDIT: Never mind, now I see it's an exception on line 3.

Re: Awk in 20 Minutes (2015)

#62

Post by j0057 »

j_z_reeves wrote: post #46

nice, I finally took the time to read the man pages for awk. And whipped out a script to count the number of errors occurred for a particular day for a postgres log file. cat logfile | awk '/ERROR:/ {counts[$1] = counts[$1] + 1}; END { for (day in counts) print day " : " counts[day]}' | sort I just needed to know how awk programs are structured, the rest is just simple programming! EDIT: I'm not sure if it's actually…

Without knowing the format of logfile, that still seems obviously correct to me.

Re: Awk in 20 Minutes (2015)

#63

Post by tannhaeuser »

arendtio wrote: post #55

Very nice conclusion of the language. However, keep in mind that awk is not the fastest language around. A pro awk thread might not be the best place to tell this story but it is fresh and true: Last weekend I was playing around with some data. At first, I thought 'let's just write a line of awk and be done with it' and so I did. The execution took 20 seconds (about 17 million lines) and everything was fine. Later th…

awk is a language with multiple implementations, so we can only talk about performance of a particular awk implementation. Which is especially true for awk because mawk (based on a vm rather than being a traditional interpreter) is so much faster than other awks if you can live with its limitations [1]. If you're using gawk, setting LANG=C also helps performance because no utf-8 honoring needs to be done. Speaking of which, I've noticed gawk on recent Ubuntus (19.10) appears broken and/or has regexp size limits breaking awk code with large regexpes (but haven't checked yet thorougly).

[1]: https://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-...

Re: Awk in 20 Minutes (2015)

#64

Post by Torwald »

known wrote: post #35
mauvehaus wrote: post #21

I learned perl in college in the 5.6 days, and did a lot of text processing with it for a time. At some point, I bit the bullet and learned awk, and I've mostly abandoned perl as a result. Why? awk is small enough that it fits in my head, or at least the bits I need every couple of months do. And if I forget, it only takes 20 minutes to put them back in. perl, by contrast, is far too large to fit in my head and comes…

Perl is Awk on steroids :)

Perl 6 is when they then took LSD and at a hang-up, they went to the clinic and came back a s two persons. One said: never again, the other: bring it on.

Re: Awk in 20 Minutes (2015)

#67

Post by arendtio »

tannhaeuser wrote: post #63
arendtio wrote: post #55

Very nice conclusion of the language. However, keep in mind that awk is not the fastest language around. A pro awk thread might not be the best place to tell this story but it is fresh and true: Last weekend I was playing around with some data. At first, I thought 'let's just write a line of awk and be done with it' and so I did. The execution took 20 seconds (about 17 million lines) and everything was fine. Later th…

awk is a language with multiple implementations, so we can only talk about performance of a particular awk implementation. Which is especially true for awk because mawk (based on a vm rather than being a traditional interpreter) is so much faster than other awks if you can live with its limitations [1]. If you're using gawk, setting LANG=C also helps performance because no utf-8 honoring needs to be done. Speaking of…

Arch Linux

  $ awk --version
  GNU Awk 5.1.0, API: 3.0 (GNU MPFR 4.0.2, GNU MP 6.2.0)

Re: Awk in 20 Minutes (2015)

#68

Post by arendtio »

wodenokoto wrote: post #57
arendtio wrote: post #55

Very nice conclusion of the language. However, keep in mind that awk is not the fastest language around. A pro awk thread might not be the best place to tell this story but it is fresh and true: Last weekend I was playing around with some data. At first, I thought 'let's just write a line of awk and be done with it' and so I did. The execution took 20 seconds (about 17 million lines) and everything was fine. Later th…

What kind of task was it? I’d imagine loading a csv-like file and doing some somewhat heavy calculations, R would win, but when I think of AWK problems I think of text manipulation, and when I think of manipulating text I do not think of R Disclaimer: I don’t actually write any AWK, but learning it is on my bucket list.

The tasks were different.

AWK -> calculate the difference between every two lines:

  awk -F ',' 'NR!=1{printf "%.0f\n", $1-ll}NR==1{print ""}{ll=$1}'
R -> Calculate the min, median, mean, and max:

  d
AFAIK, calculating the difference to its previous line for every line should be faster for data that is not sorted. I didn't try to write that one in R though.

I am sure there ways to make both things faster, I was just surprised as I didn't expect R to be faster with those two naive implementations.

Re: Awk in 20 Minutes (2015)

#69

Post by saberience »

mauvehaus wrote: post #21

I learned perl in college in the 5.6 days, and did a lot of text processing with it for a time. At some point, I bit the bullet and learned awk, and I've mostly abandoned perl as a result. Why? awk is small enough that it fits in my head, or at least the bits I need every couple of months do. And if I forget, it only takes 20 minutes to put them back in. perl, by contrast, is far too large to fit in my head and comes…

What are you using Perl/awk for that it's needed so often?

I've been a software engineer since 2005 and worked my way up to being a VP of Engineering currently and never had to use either Perl or awk (or similar). I often read about these tools on Hackernews and I find it quite mystifying as I manage to have written Java, Scala, C#, SQL, and so on for 15 years and happily never needed them.

Is this a certain kind of engineering job that requires searching through text files so often and requiring specialized tools? I've managed my whole career with ctrl-f, and highlight-all matches.

Re: Awk in 20 Minutes (2015)

#70

Post by eternauta3k »

saberience wrote: post #69
mauvehaus wrote: post #21

I learned perl in college in the 5.6 days, and did a lot of text processing with it for a time. At some point, I bit the bullet and learned awk, and I've mostly abandoned perl as a result. Why? awk is small enough that it fits in my head, or at least the bits I need every couple of months do. And if I forget, it only takes 20 minutes to put them back in. perl, by contrast, is far too large to fit in my head and comes…

What are you using Perl/awk for that it's needed so often? I've been a software engineer since 2005 and worked my way up to being a VP of Engineering currently and never had to use either Perl or awk (or similar). I often read about these tools on Hackernews and I find it quite mystifying as I manage to have written Java, Scala, C#, SQL, and so on for 15 years and happily never needed them. Is this a certain kind of…

> I've managed my whole career with ctrl-f, and highlight-all matches.

Imagine you have to do this for 100 files.

Post reply on HN