Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Learn Awk? (2016)

blog.jpalardy.com

Post reply on HN
Print view View original

91–100 of 246 posts

Re: Why Learn Awk? (2016)

#91

Post by samatman »

kazinator wrote: post #39
samatman wrote: post #32

Earlier quoted context omitted.

Well... no, it doesn't, obviously, we can take a quick look at gawk to confirm this. But, just as we can't say "awk offers the -b flag to process characters as bytes", we can't really say that cut offers any extensions not defined in the standard. An implementation could, sure. I'd prefer that it didn't, writing conformant shell scripts is hard enough.

cut -f is in the standard: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/c...

I'm quite sure we just linked to the same document, were you meaning to address the grandparent?

Re: Why Learn Awk? (2016)

#92

Post by darau1 »

bloopernova wrote: post #83

I use awk because there's an almost 100% chance that it's going to be installed on any unix system I can ssh into. I use awk because I like to visually refine my output incrementally. By combining awk with multiple other basic unix commands and pipes, I can get the data that I want out of the data I have. I'm not writing unit tests or perfect code, I'm using rough tools to do a quick one-off job. For instance, "mail…

> it's not elegant

I completely disagree.

Re: Why Learn Awk? (2016)

#93

Post by billjings »

berkes wrote: post #72

Last week I threw out AWK and replaced it with Ruby (Could've been Python, Perl or PHP even). Because AWK is not suited for CSV. Please prove me wrong! I had to parse 9million lines. Some of which contain "quoted records", others, same column, are unquoted. Some contain comma's, in the fields, most don't. CSV is like that: more like a guideline than actual sense. Two hours of googling and hacking later, I gave up and…

I had the exact same experience, but sub python for Ruby.

The lesson I took wasn't that awk sucks, though. The lesson was that CSV is not trivial, and should not be parsed with regex or string matching. It's a standard with variants, and rolling in a library will pay dividends, especially if you're parsing a wide variety of different dialects of CSV.

A related lesson I took is that once your awk script grows beyond a certain level, graduate it up to a real language. I love awk, but it excels at small scale text munging. It's not suited to anything more involved than that. If translating an awk program is a major task, then the program was already too big to begin with.

Re: Why Learn Awk? (2016)

#94

Post by rhombocombus »

bloopernova wrote: post #83

I use awk because there's an almost 100% chance that it's going to be installed on any unix system I can ssh into. I use awk because I like to visually refine my output incrementally. By combining awk with multiple other basic unix commands and pipes, I can get the data that I want out of the data I have. I'm not writing unit tests or perfect code, I'm using rough tools to do a quick one-off job. For instance, "mail…

Its ubiquity and performance open up all kinds of sophisticated data processing on a huge variety of *nix implementations. Whether it's one liners or giant data scrubs, awk is a tool that you can almost always count on having access to, even in the most restrictive or arcane environments.

Re: Why Learn Awk? (2016)

#95

Post by sigjuice »

0xff00ffee wrote: post #87
pizza234 wrote: post #57

Earlier quoted context omitted.

> I don't know of any easier equivalent of "awk '{ print $2 }'" for what it does. I'm not sure if you refer spefically to cut, but Perl has something similar and approximaly terse: > echo 'a b c' | perl -lane 'print $F[1]' Also, Perl can slice arrays, which is something that I really miss in Awk.

PERL is bloatware by comparison and less likely to be installed on distros than AWK. (e.g, embedded or slim distros. that's why you rarely see nonstandard /bin execs in shell scripts).

The awk on an embedded system is most likely a non-mainstream awk implementation with fewer or different features.

Re: Why Learn Awk? (2016)

#96

Post by 0xff00ffee »

When writing a shell script the robustness/reliabilty can be inferred from it's scope:

1. Uses shell-only commands (echo, for) - most robust; but things like basename/dirname and regex's vary by shell (sh, bash, zsh, ksh)

2. Uses /bin - might run into missing a binary but not likely, still robust and allows a richer set of tools (e.g., uname, chmod and admin-ish things live in /bin)

3. Uses /usr/bin - runs risk if missing packages, likely not very robust (packages drop things in here, like gzip, yacc, gcc)

4. Uses /usr/local/bin or /opt/local/bin - definitely requires package installs, least robust

Re: Why Learn Awk? (2016)

#97

Post by rhombocombus »

jolmg wrote: post #53
rhombocombus wrote: post #20

Earlier quoted context omitted.

While I agree that the silent failures and "opaqueness" can be off-putting, once you understand the tool and how to implement it in your workflow it is wonderfully efficient. Aside from being syntactically terse I haven't found any compelling reasons not to use it.

> Aside from being syntactically terse I haven't found any compelling reasons not to use it. For me, that's THE reason to use it. It's terseness is what allows it to be efficient enough to be used primarily interactively.

I wholeheartedly agree, but I think that is why a lot of folks get turned off from it.

Re: Why Learn Awk? (2016)

#98

Post by arminiusreturns »

bonoboTP wrote: post #64
arminiusreturns wrote: post #27

Earlier quoted context omitted.

I like to tell the story of when I was doing some genetics data wrangling and spent three days writing some perl code and I kept failing, then sent one of the researchers an email and he suggested an awk method and I turned 3 pages of perl into an awk one liner that works just fine. Now, its probably because I don't know perl very well, but as an ops type who doesnt have the classical dev/cs education, tools like it…

I find Python list/dict comprehensions, zip, range, enumerate etc. and the itertools module very good for such things. And you have immediate access to so many useful modules (csv, json, xml) and can easily extract code fragments into functions. You can also execute shell-like commands with subprocess.check_output() without ever worrying again about escaping strings or accidentally splitting them at spaces or whateve…

I agree python seems to have taken over in this space since then (7+ yrs ago) and is usually a better tool in bioinformatics, I was just using the tools I knew as an ops dude.

Re: Why Learn Awk? (2016)

#99

Post by rhombocombus »

shpeedy wrote: post #49
rhombocombus wrote: post #14

It is fast, robust, and frequently far more performant than a lot of modern tools that can be overkill for most data manipulation. I use it all the time in our ETL processes and it always works as advertised.

Perl is much faster[0], with much more features, with bunch of ready to use libraries, with package manager (CPAN), and similar syntax to awk. Why you use awk? [0]: http://rc3.org/2014/08/28/surprisingly-perl-outperforms-sed-...

I use it for a couple reasons: one, it is installed as a base app on almost every single *nix implementation on the planet, so you can count on having it even on ancient or restrictive environments (which I work in frequently); Two, awk is frequently fast enough for most needs, and generally far faster than a number of off the shelf "modern" tools. The first reason is the one that generally leads me to its use, its ubiquity and power make it a compelling tool.

Re: Why Learn Awk? (2016)

#100

Post by tgv »

berkes wrote: post #72

Last week I threw out AWK and replaced it with Ruby (Could've been Python, Perl or PHP even). Because AWK is not suited for CSV. Please prove me wrong! I had to parse 9million lines. Some of which contain "quoted records", others, same column, are unquoted. Some contain comma's, in the fields, most don't. CSV is like that: more like a guideline than actual sense. Two hours of googling and hacking later, I gave up and…

Sure, for some things awk is a good fit, for others it isn't.

But if your file is correct csv, and you use gawk, this does the trick: https://www.gnu.org/software/gawk/manual/html_node/Splitting...

Post reply on HN