Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Learn Awk? (2016)

blog.jpalardy.com

Post reply on HN
Print view View original

181–190 of 246 posts

Re: Why Learn Awk? (2016)

#181

Post by graphpapa »

nextos wrote: post #146
bloopernova wrote: post #83

I use awk because there's an almost 100% chance that it's going to be installed on any unix system I can ssh into. I use awk because I like to visually refine my output incrementally. By combining awk with multiple other basic unix commands and pipes, I can get the data that I want out of the data I have. I'm not writing unit tests or perfect code, I'm using rough tools to do a quick one-off job. For instance, "mail…

I disagree, it's quite elegant if you think in terms of relational algebra operators: * Projection (Π): awk and cut for simple cases * Selection (σ): grep for simple cases, otherwise sed & awk * Rename (ρ): sed * Set operators: join, comm...

I’m pretty mathsy but I don’t get this

Re: Why Learn Awk? (2016)

#182

Post by dredmorbius »

shmerl wrote: post #111

So why for example Awk instead of let's say Perl? The argument of "it's likely to be installed" isn't very compelling.

If you're versed in both, it comes down to taste, though there really are cases where you'll have awk (usually via busybox) but not Perl. OpenWRT comes to mind (just verified it's not present by default, though yes, packages are available).

For a huge number of simple tasks, awk is available and sufficient. It's largely a subset of Perl, so yes, there's some skills overlap, but there are times where knowing awk is the right tool and the available tool will pay off.

Re: Why Learn Awk? (2016)

#183

Post by dredmorbius »

doctor_eval wrote: post #136

It would be so great if awk had a csv mode. For whatever reason (Excel), CSV seems to be the default text format for field oriented data. Maybe I’m dumb but I’ve never come up with a separator regex that is quite right.

The simple case is:

    FS = ","
or:

    awk -F ','
If you're working with CSV data that has quoted strings with embedded commas, FPAT is your friend:

    FPAT = "([^,]+)|(\"[^\"]+\")"
See: https://www.gnu.org/software/gawk/manual/gawk.html#Splitting...

Re: Why Learn Awk? (2016)

#184

Post by ineedasername »

In the early aughts, fresh out of college, one of my first projects was to take semi-structured text from database blobs and convert it to XML. I quickly realized it was not a task to be fully automated, too many edge cases that really required human judgement because the it was only very loosely semi-structured. I turned to sed, awk, and pico. Sed & awk did the 99%, and dumped into pico when it didn't know what to do and I would resolve the issue. Doing it semi-supervised in this way was 100 times faster than doing it all manually, and 10 times faster than full automation, and probably more accurate.

But it was the ability to string together these types of command line tools that made it possible.

Re: Why Learn Awk? (2016)

#185

Post by shmerl »

dredmorbius wrote: post #182
shmerl wrote: post #111

So why for example Awk instead of let's say Perl? The argument of "it's likely to be installed" isn't very compelling.

If you're versed in both, it comes down to taste, though there really are cases where you'll have awk (usually via busybox) but not Perl. OpenWRT comes to mind (just verified it's not present by default, though yes, packages are available). For a huge number of simple tasks, awk is available and sufficient. It's largely a subset of Perl, so yes, there's some skills overlap, but there are times where knowing awk is th…

I like Perl regexes more though, especially aliases. Using \d is a lot neater than [0-9] or [[:digit:]].

Re: Why Learn Awk? (2016)

#186

Post by catalogia »

syncsynchalt wrote: post #44
cauthon wrote: post #34

Earlier quoted context omitted.

> I don't know of any easier equivalent of "awk '{ print $2 }'" for what it does. Does `cut -f2` not work? My complaint with cut is that you can't reorder columns (e.g. `cut -f3,2` ) Awk is really great for general text munging, more than just column extraction, highly recommend picking up some of the basics Edit to agree with the commenters below me: If the file isn't delimited with a single character, then cut alon…

It does not. Compare: $ echo ' 1 2 3' | cut -f2 1 2 3 $ echo ' 1 2 3' | cut -f2 -d' ' $ echo ' 1 2 3' | awk '{print $2}' 2 "-f [...] Output fields are separated by a single occurrence of the field delimiter character."

Perfect example of how "do one thing and do it well" is a lie.

Re: Why Learn Awk? (2016)

#187

Post by chaps »

3xblah wrote: post #144
jcims wrote: post #120

Earlier quoted context omitted.

I know you’re not asking for awk protips but you can prefix the block with a match condition for processing. ... | grep foo | awk ‘{print $6}’ | ... becomes ... | awk ‘/foo/{print $6}’ | ... If you start working this into your awk habits you’ll find delightful little edge cases that you can handle with other expressions before the block (you can, for example, match specific fields).

No one has mentioned changing the default field separator, e.g., awk FS=: '{print $1}' instead of cut -d: -f1 awk FS=" " '{print $1}' instead of cut -d'>' -f1

No need to explicitly set FS! Just use:

echo test,123 | awk -F, '{print $1}'

Re: Why Learn Awk? (2016)

#188

Post by shaklee3 »

0xff00ffee wrote: post #87
pizza234 wrote: post #57

Earlier quoted context omitted.

> I don't know of any easier equivalent of "awk '{ print $2 }'" for what it does. I'm not sure if you refer spefically to cut, but Perl has something similar and approximaly terse: > echo 'a b c' | perl -lane 'print $F[1]' Also, Perl can slice arrays, which is something that I really miss in Awk.

PERL is bloatware by comparison and less likely to be installed on distros than AWK. (e.g, embedded or slim distros. that's why you rarely see nonstandard /bin execs in shell scripts).

Unfortunately, for large files perl is significantly faster than awk. I was working on some very large files doing some splitting, and perl was over an order of magnitude faster.

Re: Why Learn Awk? (2016)

#189

Post by [deleted] »

danwills wrote: post #131
patrickdavey wrote: post #103

Earlier quoted context omitted.

I don't suppose your dotfiles are available anywhere. I'm just wondering what other useful things I can steal ;)

I don't have my dotfile here (on my phone) but here's some ideas from things I've aliased that I use a lot: cdd : like cd but can take a file path - : cd - (like 1-level undo for cd) .. : cd .. ... : cd ../.. (and so on up to '.......') nd : like cd but enter the dir too udd : go up (cd ..) and then rmdir xd : search for arg in parent dirs, then go as deep as possible on the other branch (like super-lazy cd, actually…

[deleted]

Re: Why Learn Awk? (2016)

#190

Post by 3xblah »

chaps wrote: post #187
3xblah wrote: post #144

Earlier quoted context omitted.

No one has mentioned changing the default field separator, e.g., awk FS=: '{print $1}' instead of cut -d: -f1 awk FS=" " '{print $1}' instead of cut -d'>' -f1

No need to explicitly set FS! Just use: echo test,123 | awk -F, '{print $1}'

Yikes. The syntax I had was wrong anyway. Should have been

   awk 'BEGIN {FS=":"};{print $1}'
One benefit of the FS variable over -F, at least in original awk, is that by using FS the delimiter can be more than one character. I guess that's why I remember FS before I remember -F. More flexible.
Post reply on HN