Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Awk in 20 Minutes (2015)

ferd.ca

Post reply on HN
Print view View original

101–110 of 128 posts

Re: Awk in 20 Minutes (2015)

#101

Post by ratsmack »

arendtio wrote: post #55

Very nice conclusion of the language. However, keep in mind that awk is not the fastest language around. A pro awk thread might not be the best place to tell this story but it is fresh and true: Last weekend I was playing around with some data. At first, I thought 'let's just write a line of awk and be done with it' and so I did. The execution took 20 seconds (about 17 million lines) and everything was fine. Later th…

I have never found anything faster then mawk. GNU awk (gawk) is orders of magnitude slower. For a proper comparison you would have to note which implementation of awk you are using.

Re: Awk in 20 Minutes (2015)

#102

Post by yiyus »

forinti wrote: post #60
mauvehaus wrote: post #21

I learned perl in college in the 5.6 days, and did a lot of text processing with it for a time. At some point, I bit the bullet and learned awk, and I've mostly abandoned perl as a result. Why? awk is small enough that it fits in my head, or at least the bits I need every couple of months do. And if I forget, it only takes 20 minutes to put them back in. perl, by contrast, is far too large to fit in my head and comes…

On the other hand, Perl is a lot faster. One small test (with a big file) I executed took 1 minute with Perl and 20 minutes with awk. And then there are those really complicated formats where awk is just not flexible enough. Awk is really useful, but it doesn't cover the same problem set as Perl does.

That depends on what you are doing. Which test did you run?

Around 10 years ago I rewrote markdown.pl in awk and it was almost 20 times faster. The speedup came from both a much faster startup time and a much faster (albeit simpler) regexp implementation.

Re: Awk in 20 Minutes (2015)

#104

Post by tejtm »

j_z_reeves wrote: post #46

nice, I finally took the time to read the man pages for awk. And whipped out a script to count the number of errors occurred for a particular day for a postgres log file. cat logfile | awk '/ERROR:/ {counts[$1] = counts[$1] + 1}; END { for (day in counts) print day " : " counts[day]}' | sort I just needed to know how awk programs are structured, the rest is just simple programming! EDIT: I'm not sure if it's actually…

j_z_reeves 9 hours ago [-]

> cat logfile | awk '/ERROR:/ {counts[$1] = counts[$1] + 1}; END { for (day in counts) print day " : " counts[day]}' | sort

Great first program! a bit less verbose could be

> awk '/ERROR:/ {counts[$1]++}END{...}' logfile

there are also ways of sorting the output but within (g)awk (asort & asorti) but sorting externally as you have is more flexible and engages another core which can be faster on large input

Re: Awk in 20 Minutes (2015)

#105

Post by yiyus »

gumby wrote: post #92

One trick I use with awk, especially throwaway ones, is to use grep to subset the data before feeding it into awk. Often the case is using awk to poke at some data to diagnose a problem, not write a script to be run often or stuck in cron. So to use an example from this nice short article: I might do grep GET log-file | awk blah-blah. Then awk doesn’t need to consider the lines I don’t care about. This is especially…

I find easier to do awk /GET/{blah-blah}

Re: Awk in 20 Minutes (2015)

#106

Post by andrewl »

saberience wrote: post #69
mauvehaus wrote: post #21

I learned perl in college in the 5.6 days, and did a lot of text processing with it for a time. At some point, I bit the bullet and learned awk, and I've mostly abandoned perl as a result. Why? awk is small enough that it fits in my head, or at least the bits I need every couple of months do. And if I forget, it only takes 20 minutes to put them back in. perl, by contrast, is far too large to fit in my head and comes…

What are you using Perl/awk for that it's needed so often? I've been a software engineer since 2005 and worked my way up to being a VP of Engineering currently and never had to use either Perl or awk (or similar). I often read about these tools on Hackernews and I find it quite mystifying as I manage to have written Java, Scala, C#, SQL, and so on for 15 years and happily never needed them. Is this a certain kind of…

The author offers one answer to this in his opening text:

I say it's useful on servers because log files, dump files, or whatever text format servers end up dumping to disk will tend to grow large, and you'll have many of them per server. If you ever get into the situation where you have to analyze gigabytes of files from 50 different servers without tools like Splunk or its equivalents, it would feel fairly bad to have to download all these files locally to then drive some forensics on them.

This personally happens to me when some Erlang nodes tend to die and leave a crash dump of 700MB to 4GB behind, or on smaller individual servers (say a VPS) where I need to quickly go through logs, looking for a common pattern.

Re: Awk in 20 Minutes (2015)

#107

Post by mrmrcoleman »

mauvehaus wrote: post #21

I learned perl in college in the 5.6 days, and did a lot of text processing with it for a time. At some point, I bit the bullet and learned awk, and I've mostly abandoned perl as a result. Why? awk is small enough that it fits in my head, or at least the bits I need every couple of months do. And if I forget, it only takes 20 minutes to put them back in. perl, by contrast, is far too large to fit in my head and comes…

I read the first line as "I learned perl in college in 5.6 days"

Re: Awk in 20 Minutes (2015)

#108

Post by DmitryOlshansky »

forinti wrote: post #60
mauvehaus wrote: post #21

I learned perl in college in the 5.6 days, and did a lot of text processing with it for a time. At some point, I bit the bullet and learned awk, and I've mostly abandoned perl as a result. Why? awk is small enough that it fits in my head, or at least the bits I need every couple of months do. And if I forget, it only takes 20 minutes to put them back in. perl, by contrast, is far too large to fit in my head and comes…

On the other hand, Perl is a lot faster. One small test (with a big file) I executed took 1 minute with Perl and 20 minutes with awk. And then there are those really complicated formats where awk is just not flexible enough. Awk is really useful, but it doesn't cover the same problem set as Perl does.

There are many AWK implementations, OSX I think have subpar AWK. GAWK was fast, last time checked.

Re: Awk in 20 Minutes (2015)

#109

Post by gumby »

yiyus wrote: post #105
gumby wrote: post #92

One trick I use with awk, especially throwaway ones, is to use grep to subset the data before feeding it into awk. Often the case is using awk to poke at some data to diagnose a problem, not write a script to be run often or stuck in cron. So to use an example from this nice short article: I might do grep GET log-file | awk blah-blah. Then awk doesn’t need to consider the lines I don’t care about. This is especially…

I find easier to do awk /GET/{blah-blah}

Oh yeah, sure! I should have been clear that when dredging a really large log file or such it can be faster to run awk on a much smaller subset when your awk program has a lot of productions. I find it easier to think "OK, just run awk on these lines that matter."

Six of one half dozen of the other.

Re: Awk in 20 Minutes (2015)

#110

Post by combatentropy »

liammonahan wrote: post #32

My absolute favorite example of what's possible with awk is this[0] calculator from Ward Cunningham about splitting expenses on a ski trip. It's a really beautiful little piece of code well-adapted to this problem. [0] - https://c2.com/doc/expense/

I glanced through the input and output. When I got to the code, I was startled by how short it was. This page is worthy of its own post on Hacker News. I am now convinced that Ward Cunningham is a genius. I was already pretty sure after reading how he invented the wiki.
Post reply on HN