Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Awk in 20 Minutes (2015)

ferd.ca

Post reply on HN
Print view View original

91–100 of 128 posts

Re: Awk in 20 Minutes (2015)

#91

Post by dan-robertson »

forinti wrote: post #60
mauvehaus wrote: post #21

I learned perl in college in the 5.6 days, and did a lot of text processing with it for a time. At some point, I bit the bullet and learned awk, and I've mostly abandoned perl as a result. Why? awk is small enough that it fits in my head, or at least the bits I need every couple of months do. And if I forget, it only takes 20 minutes to put them back in. perl, by contrast, is far too large to fit in my head and comes…

On the other hand, Perl is a lot faster. One small test (with a big file) I executed took 1 minute with Perl and 20 minutes with awk. And then there are those really complicated formats where awk is just not flexible enough. Awk is really useful, but it doesn't cover the same problem set as Perl does.

I’m a bit surprised by this. It surely depends on the use case. I tend to use awk for the same reasons as the parent (smaller to write and remember) and that the pattern-action structure works really nicely once you get used to it.

Re: Awk in 20 Minutes (2015)

#92

Post by gumby »

One trick I use with awk, especially throwaway ones, is to use grep to subset the data before feeding it into awk. Often the case is using awk to poke at some data to diagnose a problem, not write a script to be run often or stuck in cron.

So to use an example from this nice short article: I might do grep GET log-file | awk blah-blah. Then awk doesn’t need to consider the lines I don’t care about. This is especially useful when iteratively writing the awk script.

Re: Awk in 20 Minutes (2015)

#93

Post by alexhutcheson »

There are two use-cases I’ve run into where awk really shines, and is hard to replace:

1. Writing scripts for environments that only have Busybox. Technically you can write scripts in ash, but I don’t recommend it for anything beyond a couple lines. It’s missing a lot of the features from Bash that make scripting easier, and it’s easy to get mixed up if you’re used to Bash and write things that don’t work. Awk is the best scripting language available, even if you’re doing things that don’t exactly match what it was designed to do.

2. Snippets that are meant to be copy+pasted from documentation or how-to articles. In that case, it’s often not easy to distribute a separate script file, so a CLI “one-liner” is preferred. You also can’t count on Perl, Python, etc. being available on the user’s system, but awk is pretty universal.

For most other cases, I tend to create a new .py file and write a quick Python script. Even if it’s a little more overhead, it helps keep my Python skills sharp, and often it turns out that what I actually want is a little more complicated than my initial idea anyway.

Re: Awk in 20 Minutes (2015)

#94

Post by dev_tty01 »

arendtio wrote: post #55

Very nice conclusion of the language. However, keep in mind that awk is not the fastest language around. A pro awk thread might not be the best place to tell this story but it is fresh and true: Last weekend I was playing around with some data. At first, I thought 'let's just write a line of awk and be done with it' and so I did. The execution took 20 seconds (about 17 million lines) and everything was fine. Later th…

Was it a faster tool? You had a need, you tossed awk at it and accomplished your goal. Did the 20 seconds vs. 5 seconds matter? The time you spent deciding what to do and how to do it took more time. For a one-off, grab some data type of task, the tool that you know well will almost always be the fastest because you get to the end result the quickest.

I agree with the comment below. If you haven't already, try mawk instead of awk. It is often many times faster than awk (and other solutions).

Re: Awk in 20 Minutes (2015)

#95

Post by j_z_reeves »

xorcist wrote: post #74
j_z_reeves wrote: post #46

nice, I finally took the time to read the man pages for awk. And whipped out a script to count the number of errors occurred for a particular day for a postgres log file. cat logfile | awk '/ERROR:/ {counts[$1] = counts[$1] + 1}; END { for (day in counts) print day " : " counts[day]}' | sort I just needed to know how awk programs are structured, the rest is just simple programming! EDIT: I'm not sure if it's actually…

Apart from the useless use of cat, since sort does the work here something like the following would probably suffice: grep ERROR logfile | cut -f 1 -d ' ' | sort | uniq -c

yes that would also work! I forgot about the `-c` argument for uniq.

Re: Awk in 20 Minutes (2015)

#96

Post by j_z_reeves »

j0057 wrote: post #62
j_z_reeves wrote: post #46

nice, I finally took the time to read the man pages for awk. And whipped out a script to count the number of errors occurred for a particular day for a postgres log file. cat logfile | awk '/ERROR:/ {counts[$1] = counts[$1] + 1}; END { for (day in counts) print day " : " counts[day]}' | sort I just needed to know how awk programs are structured, the rest is just simple programming! EDIT: I'm not sure if it's actually…

Without knowing the format of logfile, that still seems obviously correct to me.

I was surprised by the counts[$1] = counts[$1] + 1, since I didn't think it would correctly coerce a non-existing value to a 0.

Re: Awk in 20 Minutes (2015)

#97

Post by kshacker »

saberience wrote: post #69
mauvehaus wrote: post #21

I learned perl in college in the 5.6 days, and did a lot of text processing with it for a time. At some point, I bit the bullet and learned awk, and I've mostly abandoned perl as a result. Why? awk is small enough that it fits in my head, or at least the bits I need every couple of months do. And if I forget, it only takes 20 minutes to put them back in. perl, by contrast, is far too large to fit in my head and comes…

What are you using Perl/awk for that it's needed so often? I've been a software engineer since 2005 and worked my way up to being a VP of Engineering currently and never had to use either Perl or awk (or similar). I often read about these tools on Hackernews and I find it quite mystifying as I manage to have written Java, Scala, C#, SQL, and so on for 15 years and happily never needed them. Is this a certain kind of…

So I just wrote one for personal use. I was looking for a duplicate file/photo finder and read some reviews about losing data so I said why would I trust a third party for my data and wrote a dupe finder. Use a hash (md5) to make a match (it is not perfect but it works) and then sort and print to show me the dupes. The first version was 45 lines of code although I could have reduced lines if I tried. You don't need awk strictly but I used it to prettify my output and add some filters. Current code with lots more bells and whistles is less than 150 lines (including white space) of bash and awk. And it works for me and I trust it.

I could set up a database, write multiple layers of code ... but really?

Re: Awk in 20 Minutes (2015)

#98

Post by dwheeler »

Nice intro. Awk is a useful tool when you want to do simple line-by-line processing.

Note: The article says that awk patterns can't capture groups. The standard doesn't provide that functionality, but if you use the widely-available gawk implementation, gawk does have that capability (use "match").

Re: Awk in 20 Minutes (2015)

#99

Post by ratsmack »

augustk wrote: post #56

It's also worth mentioning that local variables can be simulated using additional formal parameters. In AWK, any missing parameter in a function call is initialized to zero. Let's say we have a function CharCount which takes a character and a line of text and returns the number of occurrences of that character: function CharCount(ch, line, n) { ... } The line break in the parameter list is an AWK convention and indic…

I thought the convention was to separate local variables with three spaces like this:

    function CharCount(ch, line,   n)

Re: Awk in 20 Minutes (2015)

#100

Post by tekknolagi »

liammonahan wrote: post #32

My absolute favorite example of what's possible with awk is this[0] calculator from Ward Cunningham about splitting expenses on a ski trip. It's a really beautiful little piece of code well-adapted to this problem. [0] - https://c2.com/doc/expense/

This reminds me of a story of a guy who wrote a whole company internal debit system like this (coffee, meals out, etc) and it turned into a currency. I feel like I either saw it here, or a similar forum. Anyone have a link? My searches have not found it...
Post reply on HN