Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Learn Awk? (2016)

blog.jpalardy.com

Post reply on HN
Print view View original

191–200 of 246 posts

Re: Why Learn Awk? (2016)

#191

Post by dredmorbius »

I've been parsing some documents converted from PDF (using the Poppler library's "pdftotext" command with the "--layout" option).

I found that reading these -- sort-of half-assed structured data, but with page-chunked artefacts and idiosyncrasies -- was difficult on a line-by-line basis, and thought idly "this would be a lot easier if I could process by page instead".

Text was laid out in columns, and the amount of indenting (the whitespace between columns) was significant. So preserving this somehow would be Very Useful.

Suddenly those pesky '^L' formfeeds were an asset, not a liability. Let's treat the formfeed ("\f") as a record delimiter, and the newline ("\n") as a field delimiter. We can parse out the actual columns based on witespace, for each line:

    BEGIN { RS="\f"; FS="\n" }
    {
        pageno = NR
        lines = NF
        for( line=1; line
This gives me:

- The running tally of pages.

- Each line of the page as an individual record.

- Via the split() function, an array of columns separated by two or more spaces, which are saved as an array of gaps so I have the whitespace to play with.

Edge cases and fiddling ensue, but that's the essential bit of the code there.

Since the lines are an array, I can roll back and forth through the page (basically being able to read forward and backwards through the text record), testing values, finding out where column boundaries are, etc., and then output a page's worth of content, transposing to a single-column format, with appropriate whitespacing, when done.

In testing and debugging the output (working off of 20+ documents of 100s to ~1,000 pages), a lot of test cases, scaffolding, diagnostics, etc., have been created and removed to make sure the Right Things are happening. Easy with awk.

Re: Why Learn Awk? (2016)

#192

Post by asicsp »

3xblah wrote: post #190
chaps wrote: post #187

Earlier quoted context omitted.

No need to explicitly set FS! Just use: echo test,123 | awk -F, '{print $1}'

Yikes. The syntax I had was wrong anyway. Should have been awk 'BEGIN {FS=":"};{print $1}' One benefit of the FS variable over -F, at least in original awk, is that by using FS the delimiter can be more than one character. I guess that's why I remember FS before I remember -F. More flexible.

-F does allow multicharacter separators (at least true for me on bash shell and gawk)

    $ echo 'Sample123string42with777numbers' | awk -F'[0-9]+' '{print $2}'
    string

Re: Why Learn Awk? (2016)

#193

Post by asicsp »

emmelaich wrote: post #153
emmelaich wrote: post #147

Earlier quoted context omitted.

To pile on :-) you often want -w (match word) flag to grep. In awk, I couldn't find how to do this. I tried /\bfoo\b/ and /\ / but neither worked. I don't know why and don't care enough which brings me to my major awk irritation ... It doesn't use extended or perl REs, which makes it quite different to ruby, perl, python, java. Now, according to the man page it does ; at least on OSX (man re_format) but as mentioned…

UGH! Found the problem; it simply doesn't work. Assuming the OSX awk is the same as the freebsd awk there is a very old open bug on this: awk(1) does not support word-boundary metacharacters https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=171725

GNU awk supports \ for start and end of word anchors, which works for GNU grep/sed as well

GNU awk also supports \y which is same as \b as well as \B for opposite (same as GNU grep/sed)

Intererstingly, there's a difference between the three types of word anchors:

    $ # \b matches both start and end of word boundaries
    $ # 1st and 3rd line have space as second character
    $ echo 'I have 12, he has 2!' | grep -o '\b..\b'
    I 
    12
    , 
    he
     2

    $ # \ strictly match only start and end word boundaries respectively
    $ echo 'I have 12, he has 2!' | grep -o '\'
    12
    he

    $ # -w ensures there are no word characters around the matching text
    $ # same as: grep -oP '(?

Re: Why Learn Awk? (2016)

#194

Post by cauthon »

3xblah wrote: post #190
chaps wrote: post #187

Earlier quoted context omitted.

No need to explicitly set FS! Just use: echo test,123 | awk -F, '{print $1}'

Yikes. The syntax I had was wrong anyway. Should have been awk 'BEGIN {FS=":"};{print $1}' One benefit of the FS variable over -F, at least in original awk, is that by using FS the delimiter can be more than one character. I guess that's why I remember FS before I remember -F. More flexible.

you were close! the following works as well

  awk -v FS="\t"

Re: Why Learn Awk? (2016)

#195

Post by shiftless »

jiggawatts wrote: post #169
rectang wrote: post #10

Awk, like shell, is a fraught programming environment, full of silent failure and hidden gotchas. Even for one-liners. I use shell because I have to, not because I like it. I dread maintaining shell scripts which have a bunch of awk and sed in them. The Unix ideal of small single-purpose tools and text processing is separable from these old warhorses.

PowerShell is a lot better at this. It was designed for reading by humans, not for saving precious bytes over 300-baud terminal connections. So for example, selecting columns is easy, and can be done by name. Here's a useful little snippet demonstrating converting and processing the CSV output of the legacy "whoami" Windows command. It lists the groups a user is a member of without poking domain controllers using LDA…

Sorry, the reason I moved to OpenBSD and Linux From Scratch was to avoid getting sucked into the dark gaping asshole of Lucifer. Microsoft shitware is not allowed on my system.

Does the phrase "Embrace, Extend, Extinguish" ring a bell? Between RedHat, Ubuntu, and Poettering (in no particular order), they have already done great damage to the world of Linux. Microsoft, needless to say, is a cancer.

> I find that people that prefer terseness over verbose syntax are selfish. They simply don't care about the future maintainers of their scripts.

Nor do you, apparently, since the poor souls who follow in your footsteps will invariably be stuck within the literal hell on earth that is the Microsoft ecosystem.

Enjoy your regularly scheduled malware updates.

Re: Why Learn Awk? (2016)

#196

Post by [deleted] »

gnulinux wrote: post #134
peterwwillis wrote: post #122

Earlier quoted context omitted.

(warning, mandatory HN contrarian comment) "This is the opposite of a trend of nonsense called DevOps, where system administrators start writing unit tests and other things to help the developers warm up to them - Taco Bell Programming is about developers knowing enough about Ops (and Unix in general) so that they don't overthink things, and arrive at simple, scalable solutions" It's not possible for developers to kn…

I don't understand this devs don't understand ops nonsense. > so you waste tons of time and disk space re-getting the same pages, re-looking up the same hostnames, etc. (And that's Ops knowledge...) If it's my job to write a web scraper, it's absolutely my job to think about/solve this problem. Is this a new trend thing?

[deleted]

Re: Why Learn Awk? (2016)

#197

Post by asicsp »

LinuxBender wrote: post #7

Here are some clever one-liners for awk [1] Please be sure to add your own. [1] - https://www.commandlinefu.com/commands/matching/awk/YXdr/sor...

I have a repo dedicated for some of the cli text processing tools like grep/sed/awk/perl/sort/etc. Here's my one-liner collection for awk [1]

[1] https://github.com/learnbyexample/Command-line-text-processi...

Re: Why Learn Awk? (2016)

#198

Post by 3xblah »

cauthon wrote: post #194
3xblah wrote: post #190

Earlier quoted context omitted.

Yikes. The syntax I had was wrong anyway. Should have been awk 'BEGIN {FS=":"};{print $1}' One benefit of the FS variable over -F, at least in original awk, is that by using FS the delimiter can be more than one character. I guess that's why I remember FS before I remember -F. More flexible.

you were close! the following works as well awk -v FS="\t"

If I am not mistaken, -v is GAWK only.

Re: Why Learn Awk? (2016)

#200

Post by kortex »

rectang wrote: post #10

Awk, like shell, is a fraught programming environment, full of silent failure and hidden gotchas. Even for one-liners. I use shell because I have to, not because I like it. I dread maintaining shell scripts which have a bunch of awk and sed in them. The Unix ideal of small single-purpose tools and text processing is separable from these old warhorses.

I wish there were a set of tools like coreutils but like..."less surprising" I guess? Like I spend 98% of my focus-time in algol-likes. 1-indexing drives me batty. I always have to try a few times to craft craft cut or tail expressions to match [i:j] slice notation.

And anything more complex than that I typically just use python.

Post reply on HN