Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Awk in 20 Minutes (2015)

ferd.ca

Post reply on HN
Print view View original

51–60 of 128 posts

Re: Awk in 20 Minutes (2015)

#51

Post by chme »

asicsp wrote: post #42
chme wrote: post #40

Earlier quoted context omitted.

If you just use awk for `{ print $2 }` then I would still prefer `tr -s ' ' | cut -d ' ' -f2`, since both are part of coreutils and with `awk` you would add a additional dependency to your script.

that is not equivalent because awk will remove trailing/leading space/tab/newlines (newlines come into play with a different record separator) whereas, tr will still leave a trailing/leading space for example: $ echo ' a b c ' | tr -s ' ' | cut -d ' ' -f2 a $ echo ' a b c ' | awk '{print $2}' b and by default awk splits on space/tab/newlines, whereas in cut example above, you get only space as delimiter cut has its u…

You are right it depends on the specific use case.

It possible to use `xargs -L 1` to trim the separators, but then you would also add `findutils` as deps.

I just wanted to point out that keeping the dependencies of scripts in mind when programming is also important.

Re: Awk in 20 Minutes (2015)

#52

Post by hf »

AceJohnny2 wrote: post #25
mauvehaus wrote: post #21

I learned perl in college in the 5.6 days, and did a lot of text processing with it for a time. At some point, I bit the bullet and learned awk, and I've mostly abandoned perl as a result. Why? awk is small enough that it fits in my head, or at least the bits I need every couple of months do. And if I forget, it only takes 20 minutes to put them back in. perl, by contrast, is far too large to fit in my head and comes…

For my & others' reference (because I keep having to look it up, just like parent): perl -ne "print $_;" '-n' will run the expression over each line of the input. This is the Awk-like mode. perl -pe "s/foo/bar/" '-p' will run the expression over each line of input, and print out the (possibly modified) line. This is the Sed-like mode. Slightly more info available in `man perlrun` or https://perldoc.perl.org/perlrun.h…

One particularly mnemonic collection of switches is 'plane':

    perl -plane 'my $script'
which iterates over all files given on the command-line (or stdin) and

  + (p)rints every processed line back out
  + deals with (l)ine endings, in and out
  + (a)utosplits every line into @F
I am aware that -n and -p are mutually exclusive, but as -p overrides -n, it's seems simpler to just keep 'plane' in mind and remove the 'p' if necessary.

Re: Awk in 20 Minutes (2015)

#53

Post by chme »

em500 wrote: post #50
chme wrote: post #40

Earlier quoted context omitted.

If you just use awk for `{ print $2 }` then I would still prefer `tr -s ' ' | cut -d ' ' -f2`, since both are part of coreutils and with `awk` you would add a additional dependency to your script.

Why would being part of coreutils matter? Awk is part of POSIX, just like tr and cut. Even in very constraint environments you can count on it via busybox.

A lot of stuff is part in POSIX, but not all of that is available on every system. Also busybox configurations can vary a lot.

Re: Awk in 20 Minutes (2015)

#55

Post by arendtio »

Very nice conclusion of the language. However, keep in mind that awk is not the fastest language around. A pro awk thread might not be the best place to tell this story but it is fresh and true:

Last weekend I was playing around with some data. At first, I thought 'let's just write a line of awk and be done with it' and so I did. The execution took 20 seconds (about 17 million lines) and everything was fine.

Later that day, I came across another task which seemed too complex for an awk one-liner so I took two lines of R and was surprised when R was done within 5 seconds on the same data set.

I was happy because I found a faster tool than the one I had, but the lesson is, that just because you use a proven tool like awk, doesn't mean there aren't any better tools. Find out what works best for you.

Re: Awk in 20 Minutes (2015)

#56

Post by augustk »

It's also worth mentioning that local variables can be simulated using additional formal parameters. In AWK, any missing parameter in a function call is initialized to zero.

Let's say we have a function CharCount which takes a character and a line of text and returns the number of occurrences of that character:

    function CharCount(ch, line,
        n)
    {
        ...
    }
The line break in the parameter list is an AWK convention and indicates that n is a "local variable."

Re: Awk in 20 Minutes (2015)

#57

Post by wodenokoto »

arendtio wrote: post #55

Very nice conclusion of the language. However, keep in mind that awk is not the fastest language around. A pro awk thread might not be the best place to tell this story but it is fresh and true: Last weekend I was playing around with some data. At first, I thought 'let's just write a line of awk and be done with it' and so I did. The execution took 20 seconds (about 17 million lines) and everything was fine. Later th…

What kind of task was it? I’d imagine loading a csv-like file and doing some somewhat heavy calculations, R would win, but when I think of AWK problems I think of text manipulation, and when I think of manipulating text I do not think of R

Disclaimer: I don’t actually write any AWK, but learning it is on my bucket list.

Re: Awk in 20 Minutes (2015)

#58

Post by kqr »

liammonahan wrote: post #32

My absolute favorite example of what's possible with awk is this[0] calculator from Ward Cunningham about splitting expenses on a ski trip. It's a really beautiful little piece of code well-adapted to this problem. [0] - https://c2.com/doc/expense/

This is the sort of thing us industry professionals need to study in more depth. It doesn't have the most flexible interface, but it's human-readable, and the implementation is simple, fast, and about as obviously correct as possible.

Re: Awk in 20 Minutes (2015)

#59

Post by ggm »

em500 wrote: post #50
chme wrote: post #40

Earlier quoted context omitted.

If you just use awk for `{ print $2 }` then I would still prefer `tr -s ' ' | cut -d ' ' -f2`, since both are part of coreutils and with `awk` you would add a additional dependency to your script.

Why would being part of coreutils matter? Awk is part of POSIX, just like tr and cut. Even in very constraint environments you can count on it via busybox.

A perpetual argument is use of too many pipe separated distinct commands. The argument suggests it's lazy to use sed | awk | grep type pipe runs because in all probability sed or awk alone could have done it and you incurred two excess fork/exec() and therefore consumed kernel and userspace beyond your need. The usual rejoinder is "get nicked"

Re: Awk in 20 Minutes (2015)

#60

Post by forinti »

mauvehaus wrote: post #21

I learned perl in college in the 5.6 days, and did a lot of text processing with it for a time. At some point, I bit the bullet and learned awk, and I've mostly abandoned perl as a result. Why? awk is small enough that it fits in my head, or at least the bits I need every couple of months do. And if I forget, it only takes 20 minutes to put them back in. perl, by contrast, is far too large to fit in my head and comes…

On the other hand, Perl is a lot faster.

One small test (with a big file) I executed took 1 minute with Perl and 20 minutes with awk.

And then there are those really complicated formats where awk is just not flexible enough.

Awk is really useful, but it doesn't cover the same problem set as Perl does.

Post reply on HN