Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CLI text processing with GNU awk

learnbyexample.github.io

Post reply on HN
Print view View original

121–130 of 136 posts

Re: CLI text processing with GNU awk

#121

Post by SOLAR_FIELDS »

llimllib wrote: post #116
SOLAR_FIELDS wrote: post #87

Earlier quoted context omitted.

You don’t even need to know awk these days. Just say “how to do x munging task” in ChatGPT and you’ll get a one liner that will be just as good as if you’d say there squinting at man pages for 30 minutes

this is exactly the sort of case where you get non-portable bullshit you don't understand out of it! It spits out something that works on BSD but not on GNU, you put it in your script and _boom_ wonder why the thing blew up in prod, and oh btw you also lack the ability to debug it because you never understood it in the first place

Meh, I don’t think that is a problem endemic to shell specifically. That’s more of putting untested code in production. The thing about the fact that is a one liner is… if that happens and it’s not portable, who cares? You just turn around, paste that sucker back in, and say make it work on OsFlavor2.06 and you get back something that works. You don’t even have to fully understand why it isn’t portable, you can just ask the AI and have it explain why. If you wanted something battle tested in prod that was readable and understandable you wouldn’t be using one line shell scripts in the first place, regardless of whether they were written by an AI or not

Re: CLI text processing with GNU awk

#122

Post by btschaegg »

mbivert wrote: post #100
jmholla wrote: post #93

Earlier quoted context omitted.

One of these days I need to get around to learning awk. In the meantime, I've learned some of the deeper, stateful, features of sed. For instance, you mentioned wanting to only output a line if it was preceded by another. Here's a sed command that does so: sed -ne 'x' -e '/PREV/ {x; /CURR/ p; x}' > echo -e "PREV\nCURR\nCURR\nCURR\nPREV\nRED" | sed -ne 'x' -e '/PREV/ {x; /CURR/ p; x}' CURR This uses sed's hold buffer.…

> One of these days I need to get around to learning awk Plan9's awk(1)[0] man page provides a precise and concise (a few paragraphs) presentation of the core features of all awk implementations. Tutorials bring practical knowledge, but often lack complete and self-contained descriptions of those nifty little tools. [0]: https://man.cat-v.org/plan_9/1/awk

I still maintain that "The AWK Programming Language" [1] is one of, if not the best programming language book I've read so far.

It's short and to the point, has good examples, and cuts most of the usual fluff like "what is a variable?". Its base assumptions are: You know how to program, and you're here to learn AWK. Let's get to it.

I dearly wish there'd be more books like it for other languages.

[1]: https://archive.org/details/pdfy-MgN0H1joIoDVoIC7

Re: CLI text processing with GNU awk

#123

Post by btschaegg »

thesuperbigfrog wrote: post #109
SOLAR_FIELDS wrote: post #90

Earlier quoted context omitted.

Do all systems still come with Perl baked in these days? If so I could see reaching for that over awk/sed. If I have to install a runtime I may as well just reach for Python

>> Do all systems still come with Perl baked in these days? If you use Git for Windows ( https://gitforwindows.org/ ), it includes Perl.

…and gawk :)

Re: CLI text processing with GNU awk

#124

Post by arp242 »

ajross wrote: post #43
e63f67dd-065b wrote: post #34

Perhaps my old sysadmin hat is showing through, but I don’t quite see what the advantage of awk is over just writing the same thing in perl. I’ve seen my fair share of horrendous shell scripts from junior sysadmins, and every time I think to myself “the text processing portion would be so much cleaner in Perl”.

There aren't any technical advantages, no. Perl's features are a proper superset of awk (by design!). What's happened is that Kids Today (tm) never learned perl. So they're discovering awk as someone new to the idea of stream processing. And awk was a great idea for that, and it represented a genuine innovation worth emulating. In the late 1970's. Then of course perl did emulate and surpass it. But then got forgotten…

I'm one of those "kids these days" but did actually learn to program Perl at some point, and I generally prefer AWK. Perl is a large and complex language, I don't need it that often, and I'm not smart enough to keep remembering all of it.

Now, if I would get hired as a full-time Perl developer and spent 2 years developing Perl: it would perhaps be different. But that's not the case, and isn't for most people.

For better or worse, Perl sees a lot less usage than it once did; I rarely encounter it "in the wild" and don't even have it on my laptop because nothing needs it.

Re: CLI text processing with GNU awk

#125

Post by thesuperbigfrog »

btschaegg wrote: post #123
thesuperbigfrog wrote: post #109

Earlier quoted context omitted.

>> Do all systems still come with Perl baked in these days? If you use Git for Windows ( https://gitforwindows.org/ ), it includes Perl.

…and gawk :)

Yes. Frequently any tool set that has gawk will also include sed, perl, cut, head, tail, less, vi / vim, etc.

It is nice that Git for Windows includes bash and all these tools.

Re: CLI text processing with GNU awk

#126

Post by mplanchard »

jmholla wrote: post #93
mplanchard wrote: post #70

I love awk, and I find myself reaching for it a fair bit. One of the main things I use it for is “sed with state,” so for things like matching on a line, but only if it was preceded by some other line. I find this to be really useful for creating one-off linters, for example I made one recently to check all our migration files for CREATE INDEX without CONCURRENTLY on a particular set of very large tables where it wou…

One of these days I need to get around to learning awk. In the meantime, I've learned some of the deeper, stateful, features of sed. For instance, you mentioned wanting to only output a line if it was preceded by another. Here's a sed command that does so: sed -ne 'x' -e '/PREV/ {x; /CURR/ p; x}' > echo -e "PREV\nCURR\nCURR\nCURR\nPREV\nRED" | sed -ne 'x' -e '/PREV/ {x; /CURR/ p; x}' CURR This uses sed's hold buffer.…

This is nifty, thanks for sharing! I had no idea that sed had a hold buffer, and it's very cool that you can swap it in and out within the sed command like that. It's funny, because I went essentially the opposite way that you did: I used to know sed and awk basics, but then I properly learned awk. Since then my sed has atrophied a bit, and I still only know the basics. I'll have to run through that tutorial you linked

Re: CLI text processing with GNU awk

#127

Post by mbivert »

btschaegg wrote: post #122
mbivert wrote: post #100

Earlier quoted context omitted.

> One of these days I need to get around to learning awk Plan9's awk(1)[0] man page provides a precise and concise (a few paragraphs) presentation of the core features of all awk implementations. Tutorials bring practical knowledge, but often lack complete and self-contained descriptions of those nifty little tools. [0]: https://man.cat-v.org/plan_9/1/awk

I still maintain that "The AWK Programming Language" [1] is one of, if not the best programming language book I've read so far. It's short and to the point, has good examples, and cuts most of the usual fluff like "what is a variable?". Its base assumptions are: You know how to program, and you're here to learn AWK. Let's get to it. I dearly wish there'd be more books like it for other languages. [1]: https://archive…

I don't remember reading it but from the ToC & glimpsing at some examples/exercises, it definitely fits in the "Tutorials" category I was thinking of.

It's always delightful to see competent authors demonstrate how much sophistication is achievable in about 100 lines of simple code, by comparison with the "the Dog class inherits from the Animal class" type of examples, or "real-life" codebases. This book is definitely in the former category.

> I dearly wish there'd be more books like it for other languages.

This all reminds me of a well-known regular expression matcher[0], in about 30 lines of C, featured in "The Practice of Programming"[1].

More generally, even without dedicated books, there are common simple-but-sophisticated type of programs that are great to get to know a language, once you have basic programming skills: standard UNIX tools (cat(1), grep(1), etc.), λ-calculus interpreter, LISP interpreter, raytracer, etc. One can often find online versions serving as "solutions."

[0]: https://www.cs.princeton.edu/courses/archive/spr09/cos333/be...

[1]: https://en.wikipedia.org/wiki/The_Practice_of_Programming

Re: CLI text processing with GNU awk

#128

Post by dmd »

arp242 wrote: post #124
ajross wrote: post #43

Earlier quoted context omitted.

There aren't any technical advantages, no. Perl's features are a proper superset of awk (by design!). What's happened is that Kids Today (tm) never learned perl. So they're discovering awk as someone new to the idea of stream processing. And awk was a great idea for that, and it represented a genuine innovation worth emulating. In the late 1970's. Then of course perl did emulate and surpass it. But then got forgotten…

I'm one of those "kids these days" but did actually learn to program Perl at some point, and I generally prefer AWK. Perl is a large and complex language, I don't need it that often, and I'm not smart enough to keep remembering all of it. Now, if I would get hired as a full-time Perl developer and spent 2 years developing Perl: it would perhaps be different. But that's not the case, and isn't for most people. For bet…

Does any OS besides Windows not ship with Perl? Even on Windows, I'd assume anyone programming has WSL set up, which means you have Perl.

Re: CLI text processing with GNU awk

#129

Post by Spooky23 »

jsyolo wrote: post #91
e63f67dd-065b wrote: post #34

Perhaps my old sysadmin hat is showing through, but I don’t quite see what the advantage of awk is over just writing the same thing in perl. I’ve seen my fair share of horrendous shell scripts from junior sysadmins, and every time I think to myself “the text processing portion would be so much cleaner in Perl”.

Not being snarky, why not python over perl? what makes perl better for scripts?

By the time you figure out which env you need to be using with python, you’ll forget why you needed it.

Re: CLI text processing with GNU awk

#130

Post by Galanwe »

SOLAR_FIELDS wrote: post #120
Galanwe wrote: post #118

Earlier quoted context omitted.

More importantly it's just mega slow. The whole point of bash one liners is not to write short bash, it's to type it in your shell to get a result quickly. Just typing the url to chatgpt in my browser I would have had the time to write my one liner in she'll XD

I sincerely doubt unless you are writing shell all day every day that you will get a decently complex working one liner out faster than GPT4.

Right, but you are bending the argument to your will.

What is a "decently complex one liner"?

If it's decently complex, then it's probably not a one liner, so indeed chatgpt may be faster.

If it's a one liner, then it's probably not complex, so I would be quite confident in being faster than chatgpt.

The reality is, 99.9% of one liners are just series of pipes and filters to extract specific fields from an output, and act on it. I'm quite confident I would be faster than chatgpt for any of these. And no, I don't write bash "all day, every day".

I tend to see "every day bash" as very similar to SQL. Once you know cut, grep, find, sed and awk, even at a basic level, then you can combine them and extract pretty much anything.

Post reply on HN