Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Learn Awk? (2016)

blog.jpalardy.com

Post reply on HN
Print view View original

211–220 of 246 posts

Re: Why Learn Awk? (2016)

#211

Post by dredmorbius »

shmerl wrote: post #185
dredmorbius wrote: post #182

Earlier quoted context omitted.

If you're versed in both, it comes down to taste, though there really are cases where you'll have awk (usually via busybox) but not Perl. OpenWRT comes to mind (just verified it's not present by default, though yes, packages are available). For a huge number of simple tasks, awk is available and sufficient. It's largely a subset of Perl, so yes, there's some skills overlap, but there are times where knowing awk is th…

I like Perl regexes more though, especially aliases. Using \d is a lot neater than [0-9] or [[:digit:]].

I hear Perl may support those ;-)

(Use the right tool for the job.)

Re: Why Learn Awk? (2016)

#212

Post by doctor_eval »

gpanders wrote: post #142
doctor_eval wrote: post #136

It would be so great if awk had a csv mode. For whatever reason (Excel), CSV seems to be the default text format for field oriented data. Maybe I’m dumb but I’ve never come up with a separator regex that is quite right.

For _actually_ comma separated values, just use awk -F , '...' Is this not what you mean? Edit: This comment might be helpful to you: https://news.ycombinator.com/item?id=22110036

The problem is that the field themselves can contain quotes ('"'), which escapes the comma. So the standard FS=/-f doesn't work properly.

It looks like FPAT from your linked article is for gawk. Gawk is great, but it's not everywhere. Still - it's good to know. Thanks!

Re: Why Learn Awk? (2016)

#213

Post by emmelaich »

asicsp wrote: post #193
emmelaich wrote: post #153

Earlier quoted context omitted.

UGH! Found the problem; it simply doesn't work. Assuming the OSX awk is the same as the freebsd awk there is a very old open bug on this: awk(1) does not support word-boundary metacharacters https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=171725

GNU awk supports \ for start and end of word anchors, which works for GNU grep/sed as well GNU awk also supports \y which is same as \b as well as \B for opposite (same as GNU grep/sed) Intererstingly, there's a difference between the three types of word anchors: $ # \b matches both start and end of word boundaries $ # 1st and 3rd line have space as second character $ echo 'I have 12, he has 2!' | grep -o '\b..\b' I…

Sure, but a fair bit of the value of the tool is it's consistency across platforms.

There's no point in awk if perl etc are ubiquitous and more consistent.

Re: Why Learn Awk? (2016)

#214

Post by dredmorbius »

dredmorbius wrote: post #191

I've been parsing some documents converted from PDF (using the Poppler library's "pdftotext" command with the "--layout" option). I found that reading these -- sort-of half-assed structured data, but with page-chunked artefacts and idiosyncrasies -- was difficult on a line-by-line basis, and thought idly "this would be a lot easier if I could process by page instead". Text was laid out in columns, and the amount of i…

And to be clear: After realising this ... and knowing what to look for ... I found this documented in the GNU Awk User's guide:

https://www.gnu.org/software/gawk/manual/html_node/Multiple-...

Re: Why Learn Awk? (2016)

#215

Post by _TwoFinger »

chongli wrote: post #172
nextos wrote: post #146

Earlier quoted context omitted.

I disagree, it's quite elegant if you think in terms of relational algebra operators: * Projection (Π): awk and cut for simple cases * Selection (σ): grep for simple cases, otherwise sed & awk * Rename (ρ): sed * Set operators: join, comm...

Bravo! This is one of the most insightful comments I've read in a long time! I have been using some of these tools for years but I never thought of describing them this way. Now I can think of writing a complex query in relational algebra and translating it into these commands in a very natural way.

Here's an interesting article that links shell scripting and relational algebra - http://matt.might.net/articles/sql-in-the-shell/

Re: Why Learn Awk? (2016)

#216

Post by pizza234 »

0xff00ffee wrote: post #87
pizza234 wrote: post #57

Earlier quoted context omitted.

> I don't know of any easier equivalent of "awk '{ print $2 }'" for what it does. I'm not sure if you refer spefically to cut, but Perl has something similar and approximaly terse: > echo 'a b c' | perl -lane 'print $F[1]' Also, Perl can slice arrays, which is something that I really miss in Awk.

PERL is bloatware by comparison and less likely to be installed on distros than AWK. (e.g, embedded or slim distros. that's why you rarely see nonstandard /bin execs in shell scripts).

Even assuming the odd "bloatware" characterization, this is irrelevant. From the article's point of view of "simple tasks", bloat or not doesn't matter; what matter is the language syntax and features used to accomplish a task (and I'd add consistency across platforms).

Regarding slim/embedded distros, it depends on the use cases, and the definition of "slim". It's hard to make broad statements on their prevalence, and regardless, I've never stated that one should use Perl instead and/or that it's "better"; only stated that the option it gives is a valid one.

Re: Why Learn Awk? (2016)

#217

Post by Chris2048 »

flukus wrote: post #166
Chris2048 wrote: post #21

Hmm. I'm sure this question will induce a flamewar of practical "#NeverAwk"-ers fighting toolbelt bloat, versus tech-hoarding AWK apologists arguing against throwing something out given if fills . Here's the thing: these arguments all too commonly focus on subjective notions of "simplicity", and toy examples divorced from actual common practise, and or solid comparable benchmarks . Show me a range of practical exampl…

> Here's the thing: these arguments all too commonly focus on subjective notions of "simplicity", and toy examples divorced from actual common practise, and or solid comparable benchmarks. I think your missing the point of awk. The O'Reilly sed and awk book has some complex examples, but when I look at my own usage they are all toy examples within a much larger scope. It's more like a special DSL extension for my she…

Can you give an example of when you'd choose AWK over perl?

perl is general purpose, but that doesn't mean it can't be used for one-liners.

Re: Why Learn Awk? (2016)

#218

Post by demiol »

wahern wrote: post #163
shpeedy wrote: post #119

Earlier quoted context omitted.

Yep, but you have bug in your awk one-liner.

If you mean the lack of quotations, then the behavior is well-defined and is presumably what was intended. Per POSIX, > The print statement shall write the value of each expression argument onto the indicated output stream separated by the current output field separator (see variable OFS above), and terminated by the output record separator (see variable ORS above). The default value for OFS is and for ORS, .

> If you mean the lack of quotations,

No, lack of commas in output and broken filenames with spaces.

Re: Why Learn Awk? (2016)

#219

Post by peterwwillis »

gnulinux wrote: post #134
peterwwillis wrote: post #122

Earlier quoted context omitted.

(warning, mandatory HN contrarian comment) "This is the opposite of a trend of nonsense called DevOps, where system administrators start writing unit tests and other things to help the developers warm up to them - Taco Bell Programming is about developers knowing enough about Ops (and Unix in general) so that they don't overthink things, and arrive at simple, scalable solutions" It's not possible for developers to kn…

I don't understand this devs don't understand ops nonsense. > so you waste tons of time and disk space re-getting the same pages, re-looking up the same hostnames, etc. (And that's Ops knowledge...) If it's my job to write a web scraper, it's absolutely my job to think about/solve this problem. Is this a new trend thing?

The difference between dev & Ops is an Italian grandma vs a restaurant chef. It's different experience that gives you different knowledge and a different skillset.

Re: Why Learn Awk? (2016)

#220

Post by Annatar »

emmelaich wrote: post #147
jcims wrote: post #120

Earlier quoted context omitted.

I know you’re not asking for awk protips but you can prefix the block with a match condition for processing. ... | grep foo | awk ‘{print $6}’ | ... becomes ... | awk ‘/foo/{print $6}’ | ... If you start working this into your awk habits you’ll find delightful little edge cases that you can handle with other expressions before the block (you can, for example, match specific fields).

To pile on :-) you often want -w (match word) flag to grep. In awk, I couldn't find how to do this. I tried /\bfoo\b/ and /\ / but neither worked. I don't know why and don't care enough which brings me to my major awk irritation ... It doesn't use extended or perl REs, which makes it quite different to ruby, perl, python, java. Now, according to the man page it does ; at least on OSX (man re_format) but as mentioned…

\b is Perl RE, not ERE. AWK not only supports ERE's, but POSIX RE's as well.
Post reply on HN