Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CLI text processing with GNU awk

learnbyexample.github.io

Post reply on HN
Print view View original

101–110 of 136 posts

Re: CLI text processing with GNU awk

#101

Post by coliveira »

ajross wrote: post #53
coliveira wrote: post #50

Earlier quoted context omitted.

In the case of awk, actually yes, it is safer. The reason is that awk is a very limited language. It has only enough functionality to provide text matching and substitution. It is very difficult to use awk to do anything of high security risk, compared to a language like perl.

But awk is never used alone. You don't solve whole problems with awk, you squish it into a script with a bunch of other junk. My point is that you're making an apples-to-oranges comparison. Sure, "awk" isn't the problem, but "bash" is, and bash is undeniably a more error-prone language than perl. You surely agree with that much, right? And if you disallow "bash" for security reasons, where does that leave "awk" in th…

Just use awk for what it was designed: text search and substitution. You can run shell scripts along with awk, but that is clearly not what you should be doing if you want to design secure systems. The first rule of security is not to abuse your tools.

Re: CLI text processing with GNU awk

#103

Post by e63f67dd-065b »

whartung wrote: post #95
e63f67dd-065b wrote: post #34

Perhaps my old sysadmin hat is showing through, but I don’t quite see what the advantage of awk is over just writing the same thing in perl. I’ve seen my fair share of horrendous shell scripts from junior sysadmins, and every time I think to myself “the text processing portion would be so much cleaner in Perl”.

Awks super power, and the reason I mostly use it, is it's free read loop, free field splitting, and the pattern/condition matching model. As a LANGUAGE, it's "eh". It just happens to be "good enough". You can, of course, do all of that with Perl. But then I have to write all that boiler plate I get with awk for free. And the gains in Perls language aren't enough, for me, to dump awk. And I don't use it for "scripting…

> free read loop

perl -n

> free field splitting

perl -a ... $F[1/2/3/etc]

> pattern/condition matching model

Not quite sure what you mean, but `perl -lane 'print if /abc/'` is might be what you're looking for

The boilerplate can be mostly eliminated with the magic incantation of `perl -lane`. The trick that makes all this work is that perl defines a whole bunch of pre-defined variables and populates them with things that might be helpful (see $_, @F, etc).

Re: CLI text processing with GNU awk

#104

Post by e63f67dd-065b »

jsyolo wrote: post #91
e63f67dd-065b wrote: post #34

Perhaps my old sysadmin hat is showing through, but I don’t quite see what the advantage of awk is over just writing the same thing in perl. I’ve seen my fair share of horrendous shell scripts from junior sysadmins, and every time I think to myself “the text processing portion would be so much cleaner in Perl”.

Not being snarky, why not python over perl? what makes perl better for scripts?

Perl is much more terse for one-liners and has much more built-in for doing text processing in scripts. Stuff like implicit read loop, field separation, etc. I would say they're suitable for different jobs: if a perl script grows beyond a hundred lines (you can do a surprising amount in that space!), then Python may be the right tool.

Perl is also much more of a known target: some version of it exists on basically every single Unix, and the language really hasn't changed that much in the past decade. I have SSH'ed into multiple CentOS 6/SLES 11 (released 2009, and granted mostly to rescue data off them) servers in the past 2 years, and perl is just much more of a known target to write things against than whatever python release is on that system.

Re: CLI text processing with GNU awk

#105

Post by mplanchard »

noloblo wrote: post #85
mplanchard wrote: post #70

I love awk, and I find myself reaching for it a fair bit. One of the main things I use it for is “sed with state,” so for things like matching on a line, but only if it was preceded by some other line. I find this to be really useful for creating one-off linters, for example I made one recently to check all our migration files for CREATE INDEX without CONCURRENTLY on a particular set of very large tables where it wou…

Can you share this example of tracking state of sql with awk?

Sure! I posted a gist here, stripped of anything particular to our company: https://gist.github.com/mplanchard/07229d61bd32ce73624d9003c...

Re: CLI text processing with GNU awk

#108

Post by nmz »

martincmartin wrote: post #80
nologic01 wrote: post #61

awk one-liners are a slam dunk. The tough question whether to invest in more complex awk programming. Invariably some processing task requires more complex logic and awk provides that, but in the terse and arcane ways of early computing. Yet reaching for a modern alternative is also an overhead, may not be particularly intuitive either (hello pandas) and may even have performance issues...

For me, the big problem is libraries. Even a personal file of common functions doesn't seem that well supported, and there just doesn't seem to be a way to get third party libraries. When I start needing helper functions and splitting it into multiple lines is usually when I reach for Python instead. And then sigh, because my program will be 2 to 3 times bigger. Ruby is a great awk replacement, but unless other peopl…

The problem of libraries is namespaces, since everything is global its not worth it. and also incredibly problematic (especially since match() actually sets a 2 globals) fortunately gawk offers namespaces, it even has a flag for loading libraries from a path { gawk -i inplace } has replaced sed -i for me a couple of times. But yeah, its still lacking.

PS: This is gawk only though, but awk -f $awkmodules/mymodule.awk -f <(echo '') is an ok replacement, even though its just concatenating files

Re: CLI text processing with GNU awk

#109

Post by thesuperbigfrog »

SOLAR_FIELDS wrote: post #90
asicsp wrote: post #41

Earlier quoted context omitted.

If you are comparing Awk vs Perl for scripts, I'd prefer Perl (or Python). This post is about short one-liners for ad hoc use cases. I prefer sed/awk over Perl for such cases. Though, if you already know Perl, you could continue using it instead of having to learn more tools.

Do all systems still come with Perl baked in these days? If so I could see reaching for that over awk/sed. If I have to install a runtime I may as well just reach for Python

>> Do all systems still come with Perl baked in these days?

If you use Git for Windows (https://gitforwindows.org/), it includes Perl.

Re: CLI text processing with GNU awk

#110

Post by kazinator »

I maintain a minor side interest in Awk, along side Lisp and other things.

I developed cppawk in 2022: https://www.kylheku.com/cgit/cppawk/about/

cppawk extends Awk with preprocessing.

There is a loop macro that supports a vocabularly of clauses. Clauses can be combined for parallel and cross-product iteration. And they are user-extensible. By writing five simple macros, you can define a new clause.

Something potentially useful if you use Awk.

Cppawk is documented with multiple man pages, and covered by unit tests which run with gawk and mawk.

Post reply on HN