Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Awk in 20 Minutes (2015)

ferd.ca

Post reply on HN
Print view View original

111–120 of 128 posts

Re: Awk in 20 Minutes (2015)

#111

Post by arendtio »

tannhaeuser wrote: post #63
arendtio wrote: post #55

Very nice conclusion of the language. However, keep in mind that awk is not the fastest language around. A pro awk thread might not be the best place to tell this story but it is fresh and true: Last weekend I was playing around with some data. At first, I thought 'let's just write a line of awk and be done with it' and so I did. The execution took 20 seconds (about 17 million lines) and everything was fine. Later th…

awk is a language with multiple implementations, so we can only talk about performance of a particular awk implementation. Which is especially true for awk because mawk (based on a vm rather than being a traditional interpreter) is so much faster than other awks if you can live with its limitations [1]. If you're using gawk, setting LANG=C also helps performance because no utf-8 honoring needs to be done. Speaking of…

Since others were suggesting mawk too, I just installed it and it was indeed faster than GNU awk: The task took about 7 seconds.

Interesting enough: LANG=C increased the performance of GNU awk a bit, but actually decreased the mawk performance.

Re: Awk in 20 Minutes (2015)

#112

Post by gen220 »

mauvehaus wrote: post #85
gen220 wrote: post #22

Earlier quoted context omitted.

hehe, I hit that bump in the road too, but maybe I was too stubborn. FWIW, you can get around it. I just took a peek because I couldn't remember how to do it off the top of my head. It looks like I had to use gawk -F ',' -v FPAT='([^,]+)|("[^"/]+")' to get the behavior you're looking for. Seems like I nabbed it from https://www.gnu.org/software/gawk/manual/html_node/Splitting... Agreed that this is the sane point at…

I hadn't run across that one yet. Thank you! The risk of that seems to be if CSV allows embedded escaped quotes in a quoted string. Does it? I don't know. And CSV is pretty loosely defined. For most people it's probably "whatever Excel emits or ingests". And I think that's why we're on the same page about pulling out python and using a module where somebody has explored what the corner cases are and dealt with them f…

Yea CSV is famously bespoke, I don’t know if it’s ever been standardized?

In my experience, each implementation is more or less unique. Even if you allow escaping, different formats permit different methods of escaping and you might have to account for each! shudders

It’s much easier to support that in python than it is in awk with regexp. The awk route will eventually make you a regexp wizard, though, which confers it’s own benefits. :)

Re: Awk in 20 Minutes (2015)

#113

Post by unhammer »

forinti wrote: post #60
mauvehaus wrote: post #21

I learned perl in college in the 5.6 days, and did a lot of text processing with it for a time. At some point, I bit the bullet and learned awk, and I've mostly abandoned perl as a result. Why? awk is small enough that it fits in my head, or at least the bits I need every couple of months do. And if I forget, it only takes 20 minutes to put them back in. perl, by contrast, is far too large to fit in my head and comes…

On the other hand, Perl is a lot faster. One small test (with a big file) I executed took 1 minute with Perl and 20 minutes with awk. And then there are those really complicated formats where awk is just not flexible enough. Awk is really useful, but it doesn't cover the same problem set as Perl does.

what script, and which awk?

Re: Awk in 20 Minutes (2015)

#114

Post by DLA »

saberience wrote: post #69
mauvehaus wrote: post #21

I learned perl in college in the 5.6 days, and did a lot of text processing with it for a time. At some point, I bit the bullet and learned awk, and I've mostly abandoned perl as a result. Why? awk is small enough that it fits in my head, or at least the bits I need every couple of months do. And if I forget, it only takes 20 minutes to put them back in. perl, by contrast, is far too large to fit in my head and comes…

What are you using Perl/awk for that it's needed so often? I've been a software engineer since 2005 and worked my way up to being a VP of Engineering currently and never had to use either Perl or awk (or similar). I often read about these tools on Hackernews and I find it quite mystifying as I manage to have written Java, Scala, C#, SQL, and so on for 15 years and happily never needed them. Is this a certain kind of…

You ever deal with very large sets of large text files? You really CTRL-f through multi-GB files? Have you never needed to transform data between formats or yank out subsets of data for processing? You may have just working on business systems that did not deal with significant data volumes or data you did not control (coming from some other organization, for example).

Re: Awk in 20 Minutes (2015)

#116

Post by jftuga »

One of the things I like to do with mawk.exe under Windows is to automate the same command over a group of files.

Let's say the myPgm only takes one file name as a command line parameter, then I can so something like this:

    dir *.xyz /s/b | mawk "{print 'myPgm -x '$0}" | cmd
If the file paths have spaces in them, then you have to wrap the name within double quotes. I have found it challenging to output those in mawk -- at least easily any way. In this case, I have a windows version of tr.exe

In this case, I can do something like this:

    dir *.xyz /s/b | mawk "{print 'myPgm -x ~'$0'~'}" | tr ~ \042 | cmd
Although somewhat crude, it is effective.

Re: Awk in 20 Minutes (2015)

#118

Post by kazinator »

By the way, Awk is whitespace sensitive:

   pattern { action }  # valid

   pattern {           # valid
     action
   }

   pattern             # not what you think
   { action }          # this action has no pattern
There cannot be a newline between pattern an action. This feature allows either the pattern or the action to be omitted without ambiguity.

   pattern        # pattern with default { print } action
   { action }     # unconditional action
   pattern {      # pattern with action
      action
   }
Items can be put on a single line without ambiguity using semicolons:

   pattern ; { action } ; pattern { action }
A pattern can have multiple patterns separated by a comma. That syntax admits optional line separation after the comma separators:

   pattern,
   pattern,
   pattern {
     action
   }
The action fires by a match for any of the patterns.

The POSIX standard Awk expression grammar has no comma operator, on the other hand; the comma exists only for separating patterns, and function arguments/parameters, and in the print statement syntax.

Plus, JNIL (just now I learned). The in operator allows comma-separated expressions for testing "multi-dimensional" array membership. Here is a hello, world:

  $ awk 'BEGIN { a[1,2,3] = 4 ; print (1,2,3) in a }'
  1
(Note that multi-dimensional arrays in Awk are simulated; a string index is generated for that 1,2,3).

Re: Awk in 20 Minutes (2015)

#120

Post by marcinzelent »

Last year, I attended a tech event where Rasmus Lerdorf (the creator of PHP) was a guest speaker. When asked about his favorite programming language, he said: AWK. He didn't elaborate why but looking at this article, I think there is something to it.
Post reply on HN