Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CLI text processing with GNU awk

learnbyexample.github.io

Post reply on HN
Print view View original

111–120 of 136 posts

Re: CLI text processing with GNU awk

#111

Post by cb321 »

kazinator wrote: post #89
cb321 wrote: post #38

Of possible interest - instead of making a whole new programming language like awk, you can also just systematize generating code for an existing one with a command-line harness. This can even stay terse & keep a fairly fast edit-test turnaround in a fully statically typed language like Nim: https://github.com/c-blake/bu/blob/main/doc/rp.md

So Awk is a whole new language, but Nim isn't?

I never said Nim was unique | older than awk. While I cannot make you read my cousin comment to understand I meant "new" as clarified-"different" [1] or click through any links, I can perhaps non-redundantly emphasize that the mentioned approach "works" not just for Nim, but for any language, C & Go (impls refd in mentioned `rp.md`), and Python in another comment in this comment thread: https://news.ycombinator.com/item?id=37295399 (maybe even with `eval` there!)

Only, the approach "works" with differing levels of "success" for different use cases / contexts. It is true (whichever) shell language is still there to differ in shell 1-liner cases. That is also true of sed / awk / perl / ... If you don't want to click through on `rp.md`, you could also read Ben Hoyt's article on his Prig if you like: https://benhoyt.com/writings/prig/ discussed on HN a while back https://news.ycombinator.com/item?id=30498735

It's not actually that different from your `cppawk` that you mention elsethread.. just maybe rotated 27 degrees away in "idea space". ;-)

[1] https://news.ycombinator.com/item?id=37293475

Re: CLI text processing with GNU awk

#112

Post by thesuperbigfrog »

sureglymop wrote: post #57
ajross wrote: post #43

Earlier quoted context omitted.

There aren't any technical advantages, no. Perl's features are a proper superset of awk (by design!). What's happened is that Kids Today (tm) never learned perl. So they're discovering awk as someone new to the idea of stream processing. And awk was a great idea for that, and it represented a genuine innovation worth emulating. In the late 1970's. Then of course perl did emulate and surpass it. But then got forgotten…

Why is that cringe? They genuinely probably came across awk before perl (I know I did, I read "The AWK Programming Language" and then went on to "The C Programming Language"). Having that said, awk is great and it's been the same for decades and available on every system (the same can't really be said about perl).

>> awk is great and it's been the same for decades and available on every system (the same can't really be said about perl).

The only issue with AWK is that there are many implementations and they are not always compatible with one another:

https://www.gnu.org/software/gawk/manual/html_node/Other-Ver...

I have ported AWK scripts from legacy Unix systems to Linux and ran into incompatibilities that required some adjustments to the scripts.

Curious: what systems have AWK, but do not have Perl?

Re: CLI text processing with GNU awk

#113

Post by reddit_clone »

jsyolo wrote: post #91
e63f67dd-065b wrote: post #34

Perhaps my old sysadmin hat is showing through, but I don’t quite see what the advantage of awk is over just writing the same thing in perl. I’ve seen my fair share of horrendous shell scripts from junior sysadmins, and every time I think to myself “the text processing portion would be so much cleaner in Perl”.

Not being snarky, why not python over perl? what makes perl better for scripts?

Perl is a progression of that particular environment. It is a superset of shell/grep/awk/sed.

A shell command works exactly as you would expect copied literally inside a backquote. With all the other goodies of a real programming langauge.

Doing this in Python (to me atleast) seems unnatural.

Re: CLI text processing with GNU awk

#114

Post by thesuperbigfrog »

coliveira wrote: post #52
qalmakka wrote: post #13

Awk is fine and dandy but, like wity Sed, I think that it's almost always replaceable with Perl which is way nicer to use, and ubiquitous. Every OS (except Windows) I laid my hands on in the last 15 years has had Perl installed in either its default install or pulled in as a dependency almost immediately (a LOT of stuff depends on Perl in any Unix system). This is, unless you are running on an embedded environment, b…

The difference is that learning pearl is an ordeal that will take several weeks at the minimum. Learning awk can be done in one afternoon, after reading a man page and a few examples. And it really works for the tasks it was designed. So I think awk is superior to perl for the purpose it was created.

>> The difference is that learning pearl is an ordeal that will take several weeks at the minimum.

I am not sure about pearl, but Perl is not that different from most other programming languages. If you are familiar with Javascript or Python, learning the basics of Perl is pretty easy:

https://perldoc.perl.org/perlintro

https://www.perltutorial.org/

Perl is designed for text processing, so it has a powerful regular expression engine. Writing regular expressions can be difficult, but it is a great skill to have in your toolkit.

Fun Fact: If the programming language you are using has support for regular expressions, they are almost certainly Perl-compatible regular expressions because Perl's regular expression syntax is more widely used and more popular than other regular expression syntaxes (e.g. POSIX, etc.).

Re: CLI text processing with GNU awk

#115

Post by llimllib »

version_five wrote: post #62
llimllib wrote: post #58

Earlier quoted context omitted.

You can do it with cut too: $ echo "key: value" | cut -wf 2 value but whether it's actually "simpler" is open to debate edit: actually gnu cut lacks -w, so this is bsd-only. lol computers, stick with awk

What does -w do? This works without it, no? Edit, found it, "use whitespace as the delimiter" https://www.unix.com/man-page/FreeBSD/1/cut/ For most cases like the OP you'd know the delimiter anyway so I don't think the absence is a big deal, and if not it would be easy to use tr or sed to make it consistent

the important thing is that it uses _consecutive_ whitespace as a the delimiter, so you'd have to use sed to collapse all the whitespace down to one tab.

At that point, awk is vastly simpler

Re: CLI text processing with GNU awk

#116

Post by llimllib »

SOLAR_FIELDS wrote: post #87
rigelina wrote: post #84

Earlier quoted context omitted.

I can't tell you how many times I pipe in rev to put my text where I want it for cut (then rev it again). Abbreviated example, getting the service names from a k8s cluster looks roughly like (actual command does a bit more processing): kubectl get deployments -o wide | rev | cut -d'=' -f1 | rev But if it's just gobbling whitespace, xargs without a command can be your friend. $ echo "key: value" | cut -d: -f2 | xargs…

You don’t even need to know awk these days. Just say “how to do x munging task” in ChatGPT and you’ll get a one liner that will be just as good as if you’d say there squinting at man pages for 30 minutes

this is exactly the sort of case where you get non-portable bullshit you don't understand out of it! It spits out something that works on BSD but not on GNU, you put it in your script and _boom_ wonder why the thing blew up in prod, and oh btw you also lack the ability to debug it because you never understood it in the first place

Re: CLI text processing with GNU awk

#117

Post by kagevf »

SOLAR_FIELDS wrote: post #86
nologic01 wrote: post #61

awk one-liners are a slam dunk. The tough question whether to invest in more complex awk programming. Invariably some processing task requires more complex logic and awk provides that, but in the terse and arcane ways of early computing. Yet reaching for a modern alternative is also an overhead, may not be particularly intuitive either (hello pandas) and may even have performance issues...

I used to not like them pre chat gpt. But nowadays when you can paste an arcane awk/sed illegible one liner into an AI and have it describe step by step what it’s doing in totally fine with it now. I still don’t like them as much as a few lines of python for unit test ability reasons but sometimes you just straight up don’t need unit tests for some quick data munging task

> have it describe step by step what it’s doing

And then building on the original “update the script to do $thing” where $thing isn’t obvious/trivial. It saves a lot of time.

Re: CLI text processing with GNU awk

#118

Post by Galanwe »

llimllib wrote: post #116
SOLAR_FIELDS wrote: post #87

Earlier quoted context omitted.

You don’t even need to know awk these days. Just say “how to do x munging task” in ChatGPT and you’ll get a one liner that will be just as good as if you’d say there squinting at man pages for 30 minutes

this is exactly the sort of case where you get non-portable bullshit you don't understand out of it! It spits out something that works on BSD but not on GNU, you put it in your script and _boom_ wonder why the thing blew up in prod, and oh btw you also lack the ability to debug it because you never understood it in the first place

More importantly it's just mega slow.

The whole point of bash one liners is not to write short bash, it's to type it in your shell to get a result quickly.

Just typing the url to chatgpt in my browser I would have had the time to write my one liner in she'll XD

Re: CLI text processing with GNU awk

#119

Post by vbrandl »

rascul wrote: post #88
coliveira wrote: post #50

Earlier quoted context omitted.

In the case of awk, actually yes, it is safer. The reason is that awk is a very limited language. It has only enough functionality to provide text matching and substitution. It is very difficult to use awk to do anything of high security risk, compared to a language like perl.

> It has only enough functionality to provide text matching and substitution. Gawk at least can do a lot more than that. Reading and writing files, network communications, and run arbitrary shell commands, for example. It's certainly not as powerful as perl but it's also not limited to just text matching and substitution. Edit: figured I would provide some examples. Here's an http server and a first person shooter in…

There is a virus written in awk that infects other awk scripts[0]. And according to wikipedia, the language is Turing complete.

[0]: https://github.com/SPTHvx/ezines/tree/main/dc5/CODES/Perfori...

Re: CLI text processing with GNU awk

#120

Post by SOLAR_FIELDS »

Galanwe wrote: post #118
llimllib wrote: post #116

Earlier quoted context omitted.

this is exactly the sort of case where you get non-portable bullshit you don't understand out of it! It spits out something that works on BSD but not on GNU, you put it in your script and _boom_ wonder why the thing blew up in prod, and oh btw you also lack the ability to debug it because you never understood it in the first place

More importantly it's just mega slow. The whole point of bash one liners is not to write short bash, it's to type it in your shell to get a result quickly. Just typing the url to chatgpt in my browser I would have had the time to write my one liner in she'll XD

I sincerely doubt unless you are writing shell all day every day that you will get a decently complex working one liner out faster than GPT4.
Post reply on HN