awk one-liners are a slam dunk. The tough question whether to invest in more complex awk programming. Invariably some processing task requires more complex logic and awk provides that, but in the terse and arcane ways of early computing. Yet reaching for a modern alternative is also an overhead, may not be particularly intuitive either (hello pandas) and may even have performance issues...
CLI text processing with GNU awk
61–70 of 136 posts
Re: CLI text processing with GNU awk
#6299.9% of my awk use case is to split a line (a la "cut - d\ - f) while discarding successive spaces. e.g.: $ echo "key: value" | awk '{print $1}' value Open to a simpler replacement :-)
You can do it with cut too: $ echo "key: value" | cut -wf 2 value but whether it's actually "simpler" is open to debate edit: actually gnu cut lacks -w, so this is bsd-only. lol computers, stick with awk
What does -w do? This works without it, no?
Edit, found it, "use whitespace as the delimiter"
https://www.unix.com/man-page/FreeBSD/1/cut/
For most cases like the OP you'd know the delimiter anyway so I don't think the absence is a big deal, and if not it would be easy to use tr or sed to make it consistent
Re: CLI text processing with GNU awk
#63Earlier quoted context omitted.
In the case of awk, actually yes, it is safer. The reason is that awk is a very limited language. It has only enough functionality to provide text matching and substitution. It is very difficult to use awk to do anything of high security risk, compared to a language like perl.
But awk is never used alone. You don't solve whole problems with awk, you squish it into a script with a bunch of other junk. My point is that you're making an apples-to-oranges comparison. Sure, "awk" isn't the problem, but "bash" is, and bash is undeniably a more error-prone language than perl. You surely agree with that much, right? And if you disallow "bash" for security reasons, where does that leave "awk" in th…
You’re right. But the alternative might be bash + perl or just bash. Or none of them. Perl is anyway the first one to go.
Re: CLI text processing with GNU awk
#64Of possible interest - instead of making a whole new programming language like awk, you can also just systematize generating code for an existing one with a command-line harness. This can even stay terse & keep a fairly fast edit-test turnaround in a fully statically typed language like Nim: https://github.com/c-blake/bu/blob/main/doc/rp.md
If you think that's a good idea, you don't understand why tools like awk/perl/sed/etc exist and are popular. They are, by design, optimized toward specific kinds of use cases.
In fact, their dynamically typed nature is a perfect example of that since it's much easier to quickly manipulate strings in a language that isn't so strict, as they'll do more heavy lifting for you via automatic coercion while limiting extra syntax/boilerplate (which, granted, is less of a problem with modern type inference). That makes it a lot easier to toss together quick one-liners and glue code, which is where these tools shine in the first place.
Hell, even something like python or ruby is just a little too structured for my taste when doing something quick and dirty, which is why I love perl as it can be unstructured if that's all I need, or I can create a more structured program if that's what the problem requires.
Re: CLI text processing with GNU awk
#6599.9% of my awk use case is to split a line (a la "cut - d\ - f) while discarding successive spaces. e.g.: $ echo "key: value" | awk '{print $1}' value Open to a simpler replacement :-)
Check out https://github.com/sstadick/hck and https://github.com/theryangeary/choose - both are alternatives for cut/awk and allows regex based split as well. Though, they don't remove starting/ending whitespaces IIRC.
I wrote a script (https://github.com/learnbyexample/regexp-cut) that uses `awk` to provide a `cut`-like tool with regex-based split, negative index, etc. And this will take care of starting/ending whitespaces as that's the default `awk` behavior.
Re: CLI text processing with GNU awk
#66I have been using ChatGPT for generating these kind of small CLI like this. My prompts look like this: - use jq to count a nested array "a.b.c.d" - find and delete empty folders using `find` - find and replace text using sed/awk I found that using ChatGPT for these purposes boosted my productivity tremendously.
My usages of tools like awk, sed and Bash scripting has increased an enormous amount thanks to ChatGPT/GPT-4.
I'm using those on a weekly basis now, because I don't have to memorize details of entirely new programming languages in order to apply them to small problems.
Smaller languages that I never took the time to learn are no longer something I avoid. I even use AppleScript now! https://til.simonwillison.net/gpt3/chatgpt-applescript
Re: CLI text processing with GNU awk
#67Earlier quoted context omitted.
There aren't any technical advantages, no. Perl's features are a proper superset of awk (by design!). What's happened is that Kids Today (tm) never learned perl. So they're discovering awk as someone new to the idea of stream processing. And awk was a great idea for that, and it represented a genuine innovation worth emulating. In the late 1970's. Then of course perl did emulate and surpass it. But then got forgotten…
Awk is a useful language that you can learn in one afternoon, after reading the man page and a few examples. And then you can spend your whole life using it for several projects. You cannot do that with perl. That's why awk has a longer shelf life than perl.
> And then you can spend your whole life using it for several projects. You cannot do that with perl. That's why awk has a longer shelf life than perl.
A Perl developer would of course say you have this completely backwards and even if I haven't programmed Perl much, or even at all for the last decade I would tend to agree.
Re: CLI text processing with GNU awk
#68Earlier quoted context omitted.
ChatGPT is a great time saver for those who already know how to use awk. But it should not be used by those who are unfamiliar. Just an example, I saw someone come up with a great awk line to change some text in a nested directory. He then pasted into bash. Only once the server went down did anybody realize that he forgot to cd into the proper directory and he wiped out not only the server config but also all the use…
That's not really a ChatGPT issue, people pasting in slightly wrong commands (or right commands in the wrong folder) is a tale as old as time - well, as old as linux at least. Short of saying that nobody who's already an expert should ever touch a CLI, the lesson from that story is "be as careful as possible, then be more careful, and also have backups of everything" not "don't use a LLM to help".
Yeah, that exact same problem could easily affect someone who spent hours cobbling together the same awk script from Google searches and StackOverflow.
Re: CLI text processing with GNU awk
#69Earlier quoted context omitted.
I use Gumroad/Leanpub to sell my ebooks. As far as I know, they don't support the "choose to pay after" model. You can see the number of paid sales for the bundles under the "I want this!" button. When the price is 0, it shows the total of both paid/free users. I started selling ebooks about 5 years back. Where I live, my monthly living cost is just $150. While the first two years of sales were just about enough to c…
> Where I live, my monthly living cost is just $150. That's quite low! Mind if I ask where in the world that is?
Outskirts of a second-tier city in southern India. I live a modest lifestyle - no vehicles, desktop instead of laptop, live alone etc.
Re: CLI text processing with GNU awk
#70I love awk, and I find myself reaching for it a fair bit. One of the main things I use it for is “sed with state,” so for things like matching on a line, but only if it was preceded by some other line. I find this to be really useful for creating one-off linters, for example I made one recently to check all our migration files for CREATE INDEX without CONCURRENTLY on a particular set of very large tables where it would cause issues. Since sql statements can be spread over multiple lines, it was difficult to write a straightforward match, but awk can track state like “I’m in a create statement,” “I’m creating an index,” etc. across multiple lines, which allowed me to cobble together something that has worked well for about a year now.