Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CLI text processing with GNU awk

learnbyexample.github.io

Post reply on HN
Print view View original

81–90 of 136 posts

Re: CLI text processing with GNU awk

#81

Post by cb321 »

BaseballPhysics wrote: post #64
cb321 wrote: post #38

Of possible interest - instead of making a whole new programming language like awk, you can also just systematize generating code for an existing one with a command-line harness. This can even stay terse & keep a fairly fast edit-test turnaround in a fully statically typed language like Nim: https://github.com/c-blake/bu/blob/main/doc/rp.md

If you think that's a good idea, you don't understand why tools like awk/perl/sed/etc exist and are popular. They are, by design, optimized toward specific kinds of use cases. In fact, their dynamically typed nature is a perfect example of that since it's much easier to quickly manipulate strings in a language that isn't so strict, as they'll do more heavy lifting for you via automatic coercion while limiting extra s…

It's just a different & in my experience often neglected point in a similar design space (as that initial, linked text argues). Your tastes & use cases are your own. Almost everything "all depends" upon so very much in computer systems & in life.

To add some more color, Nim is also a very adaptable prog.lang. I believe there are converts from Perl in its fan base. Nim's creator long ago recreated some Perl in Nim: https://nim-lang.org/araq/perlish.html

Anyway, it's a different set of trade-offs to consider which I thought some reading about learning awk with open minds might find interesting. That's all, really.

Re: CLI text processing with GNU awk

#83

Post by cb321 »

martincmartin wrote: post #80
nologic01 wrote: post #61

awk one-liners are a slam dunk. The tough question whether to invest in more complex awk programming. Invariably some processing task requires more complex logic and awk provides that, but in the terse and arcane ways of early computing. Yet reaching for a modern alternative is also an overhead, may not be particularly intuitive either (hello pandas) and may even have performance issues...

For me, the big problem is libraries. Even a personal file of common functions doesn't seem that well supported, and there just doesn't seem to be a way to get third party libraries. When I start needing helper functions and splitting it into multiple lines is usually when I reach for Python instead. And then sigh, because my program will be 2 to 3 times bigger. Ruby is a great awk replacement, but unless other peopl…

I'd bet you could do a harness like https://news.ycombinator.com/item?id=37292882 mentions but with Python in like an hour and then you could stay in both one syntax and more significantly in one library ecosystem. Why, 3 such things may even already exist. :) The syntax/semantics is not as optimized for 1-liner brevity, but everything has trade-offs.

Re: CLI text processing with GNU awk

#84

Post by rigelina »

llimllib wrote: post #58
Galanwe wrote: post #54

99.9% of my awk use case is to split a line (a la "cut - d\ - f) while discarding successive spaces. e.g.: $ echo "key: value" | awk '{print $1}' value Open to a simpler replacement :-)

You can do it with cut too: $ echo "key: value" | cut -wf 2 value but whether it's actually "simpler" is open to debate edit: actually gnu cut lacks -w, so this is bsd-only. lol computers, stick with awk

I can't tell you how many times I pipe in rev to put my text where I want it for cut (then rev it again).

Abbreviated example, getting the service names from a k8s cluster looks roughly like (actual command does a bit more processing):

kubectl get deployments -o wide | rev | cut -d'=' -f1 | rev

But if it's just gobbling whitespace, xargs without a command can be your friend.

$ echo "key: value" | cut -d: -f2 | xargs

value

My brain generally goes "rev sed head tail xargs cut tr ... screw it, I'll use python ... someday I shall learn awk." There's a young engineer on my team that knows awk, and I'm envious.

Re: CLI text processing with GNU awk

#85

Post by noloblo »

mplanchard wrote: post #70

I love awk, and I find myself reaching for it a fair bit. One of the main things I use it for is “sed with state,” so for things like matching on a line, but only if it was preceded by some other line. I find this to be really useful for creating one-off linters, for example I made one recently to check all our migration files for CREATE INDEX without CONCURRENTLY on a particular set of very large tables where it wou…

Can you share this example of tracking state of sql with awk?

Re: CLI text processing with GNU awk

#86

Post by SOLAR_FIELDS »

nologic01 wrote: post #61

awk one-liners are a slam dunk. The tough question whether to invest in more complex awk programming. Invariably some processing task requires more complex logic and awk provides that, but in the terse and arcane ways of early computing. Yet reaching for a modern alternative is also an overhead, may not be particularly intuitive either (hello pandas) and may even have performance issues...

I used to not like them pre chat gpt. But nowadays when you can paste an arcane awk/sed illegible one liner into an AI and have it describe step by step what it’s doing in totally fine with it now. I still don’t like them as much as a few lines of python for unit test ability reasons but sometimes you just straight up don’t need unit tests for some quick data munging task

Re: CLI text processing with GNU awk

#87

Post by SOLAR_FIELDS »

rigelina wrote: post #84
llimllib wrote: post #58

Earlier quoted context omitted.

You can do it with cut too: $ echo "key: value" | cut -wf 2 value but whether it's actually "simpler" is open to debate edit: actually gnu cut lacks -w, so this is bsd-only. lol computers, stick with awk

I can't tell you how many times I pipe in rev to put my text where I want it for cut (then rev it again). Abbreviated example, getting the service names from a k8s cluster looks roughly like (actual command does a bit more processing): kubectl get deployments -o wide | rev | cut -d'=' -f1 | rev But if it's just gobbling whitespace, xargs without a command can be your friend. $ echo "key: value" | cut -d: -f2 | xargs…

You don’t even need to know awk these days. Just say “how to do x munging task” in ChatGPT and you’ll get a one liner that will be just as good as if you’d say there squinting at man pages for 30 minutes

Re: CLI text processing with GNU awk

#88

Post by rascul »

coliveira wrote: post #50
ajross wrote: post #42

Earlier quoted context omitted.

Sorry, but that's ridiculous. Any general purpose programming language is a vector for bugs and security problems, but come on: you're genuinely trying to say that a kludge of bash+sed+awk is objectively more "secure" than a single perl script to solve the same problem?

In the case of awk, actually yes, it is safer. The reason is that awk is a very limited language. It has only enough functionality to provide text matching and substitution. It is very difficult to use awk to do anything of high security risk, compared to a language like perl.

> It has only enough functionality to provide text matching and substitution.

Gawk at least can do a lot more than that. Reading and writing files, network communications, and run arbitrary shell commands, for example. It's certainly not as powerful as perl but it's also not limited to just text matching and substitution.

Edit: figured I would provide some examples. Here's an http server and a first person shooter in gawk. Maybe not so practical but they show some of gawk's capabilities.

https://github.com/kevin-albert/awkserver

https://github.com/TheMozg/awk-raycaster

Re: CLI text processing with GNU awk

#89

Post by kazinator »

cb321 wrote: post #38

Of possible interest - instead of making a whole new programming language like awk, you can also just systematize generating code for an existing one with a command-line harness. This can even stay terse & keep a fairly fast edit-test turnaround in a fully statically typed language like Nim: https://github.com/c-blake/bu/blob/main/doc/rp.md

So Awk is a whole new language, but Nim isn't?

Re: CLI text processing with GNU awk

#90

Post by SOLAR_FIELDS »

asicsp wrote: post #41
e63f67dd-065b wrote: post #34

Perhaps my old sysadmin hat is showing through, but I don’t quite see what the advantage of awk is over just writing the same thing in perl. I’ve seen my fair share of horrendous shell scripts from junior sysadmins, and every time I think to myself “the text processing portion would be so much cleaner in Perl”.

If you are comparing Awk vs Perl for scripts, I'd prefer Perl (or Python). This post is about short one-liners for ad hoc use cases. I prefer sed/awk over Perl for such cases. Though, if you already know Perl, you could continue using it instead of having to learn more tools.

Do all systems still come with Perl baked in these days? If so I could see reaching for that over awk/sed. If I have to install a runtime I may as well just reach for Python
Post reply on HN