Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Awk: The Power and Promise of a 40-Year-Old Language

fosslife.org

Post reply on HN
Print view View original

51–60 of 122 posts

Re: Awk: The Power and Promise of a 40-Year-Old Language

#51

Post by throwawayboise »

jfk13 wrote: post #16
Scarbutt wrote: post #9

Earlier quoted context omitted.

Yes but you can't learn perl as quickly as you can learn awk.

Though you can learn just enough perl to do awk-like things fairly easily. And then grow from there as needed.

IDK. On my OpenBSD system the awk man page is under 500 lines, and it pretty much covers the subject.

I've tried to get started in Perl a few times, and just found it weird. It doesn't click. Awk is kind of weird too but it's so simple it doesn't matter.

I'm sure I would eventually get Perl if I had to use it. But for me, awk and sed and shell scripting have covered my needs.

Re: Awk: The Power and Promise of a 40-Year-Old Language

#52

Post by throwawayboise »

justin_oaks wrote: post #7

I only recently learned Awk enough to be useful. But I still don't reach for it when I probably should. What are the most common cases where you reach for Awk instead of some other tools? I recently used it to parse and recombine data from the OpenVPN status file. That file has a few differently formatted tables in the same file. Using Awk, I was able to change a variable as each table was encountered, this I could c…

I use it a lot to filter, slice, and dice CSV (or other delimited) or fixed-format files. Sometimes I'll use q[1] if my needs are more complex. Or awk piped to q. It can be used as a fairly decent report generator for plain-text or HTML reports.

An time I want to process a bunch of lines in a text file, awk is my first consideration.

[1] http://harelba.github.io/q/

Re: Awk: The Power and Promise of a 40-Year-Old Language

#53

Post by klyrs »

dunefox wrote: post #37
vyuh wrote: post #13

"A good programmer uses the most powerful tool to do a job. A great programmer uses the least powerful tool that does the job." I believe this, and I always try to find the combination of simple and lightweight tools which does the job at hand correctly. Awk sometimes proves surprisingly powerful. Just look at the concision of this awk one liner doing a fairly complex job: zcat large.log.gz | awk '{print $0 | "gzip -…

Ehh. Until the 'job' gets extended and then your simple tool makes it exponentially more complex and you have to rewrite it with the more powerful tool.

The nice thing about a 1-liner is you only lose a few minutes to throwing it out entirely and rewriting it to fit a new purpose. Dwelling on what might be needed is of limited utility, because of the very real possibility that what's actually needed in the future is wildly different from what you spent all that time planning for.

Re: Awk: The Power and Promise of a 40-Year-Old Language

#54

Post by michaelcampbell »

cogman10 wrote: post #49
phkahler wrote: post #25

I never use Awk until last year. I wanted to monitor an embedded device with little more than bustbox and python on it. There was quite a bit of information in the log files (I had already written a custom log file viewer with some highlighting) but I wanted to monitor in real-time. Somehow I decided to use Awk to monitor the tail of the log file and do realtime bar-graphs by generating appropriate cursor control seq…

The biggest reason to learn AWK, IMO, is that it's on pretty much every single linux distribution. You might not have perl or python. You WILL have AWK. Only the most minimal of minimal linux systems will exclude it. Even busybox includes awk. That's how essential it's viewed.

I'm curious what linux distros don't have either some version of perl or python.

I like awk, mind, but this is not necessarily (IME) a good argument for it.

Re: Awk: The Power and Promise of a 40-Year-Old Language

#55

Post by throwawayboise »

chasil wrote: post #28
exdsq wrote: post #23

Earlier quoted context omitted.

I had to take large CSV files like {question, right_ans, wrong_ans1, wrong_ans2, wrong_ans3} and covert them into SQL insert files. Few caveats - some could be duplicates, some characters were not allowed, and some had formatting issues. The first issue was avoided by upserting, but the other two I used Awk and Sed for and put together a fairly robust script far quicker than if I reached for Python. I probably would…

Awk is not really very good at reading complex CSVs (as defined in RFC-4180), where newlines (record separators) can appear within quoted strings. It can be done, but sometimes it's tricky. The PHP fgetcsv function has been more convenient when I have had more exotic examples. If the CSV is simple, awk remains a very good tool.

CSVs with quoted fields and imbedded newlines can be troublesome in awk. Years ago I had found a script that worked for me, I'm not sure but I think it was this:

http://lorance.freeshell.org/csv/

There's also https://github.com/dbro/csvquote which is more unix-like in philosophy: it sits in a pipeline, and only handles transforming the CVS data into something that awk (or other utilities) can more easily deal with. I haven't used it but will probably try it next time I need something like that.

Re: Awk: The Power and Promise of a 40-Year-Old Language

#56

Post by chasil »

throwawayboise wrote: post #50
chasil wrote: post #46

Earlier quoted context omitted.

OpenBSD's binary package system is written in perl.

Probably as much for legacy reasons as anything else. Perl was the chosen scripting language for utilities, it works, they understand it, and they've kept with it. Sort of how they stay with CVS for their source repository. Python isn't even installed on a base OpenBSD system.

Mark Espie rewrote the entire package system in perl in 2010, which is a bit late to be classed as legacy.

https://undeadly.org/cgi?action=article;sid=20100323141307

I'm not sure what was used for the version before this, but the original BSD package system was written in C.

Re: Awk: The Power and Promise of a 40-Year-Old Language

#57

Post by [deleted] »

phkahler wrote: post #25

I never use Awk until last year. I wanted to monitor an embedded device with little more than bustbox and python on it. There was quite a bit of information in the log files (I had already written a custom log file viewer with some highlighting) but I wanted to monitor in real-time. Somehow I decided to use Awk to monitor the tail of the log file and do realtime bar-graphs by generating appropriate cursor control seq…

[deleted]

Re: Awk: The Power and Promise of a 40-Year-Old Language

#58

Post by aidenn0 »

rakoo wrote: post #43
vyuh wrote: post #13

"A good programmer uses the most powerful tool to do a job. A great programmer uses the least powerful tool that does the job." I believe this, and I always try to find the combination of simple and lightweight tools which does the job at hand correctly. Awk sometimes proves surprisingly powerful. Just look at the concision of this awk one liner doing a fairly complex job: zcat large.log.gz | awk '{print $0 | "gzip -…

If I understand this correctly, it will gzip every line separately instead of gzipping them together... it's not really the most effective but it does work

It does not. The pipe command leaves the pipe open and successive pipes with identical strings remain open until the pipe is explicitly closed.

[edit]

Here's the link to the gawk documentation, but most flavors of AWK work similarly: https://www.gnu.org/software/gawk/manual/gawk.html#Close-Fil...

Re: Awk: The Power and Promise of a 40-Year-Old Language

#60

Post by rakoo »

aidenn0 wrote: post #58
rakoo wrote: post #43

Earlier quoted context omitted.

If I understand this correctly, it will gzip every line separately instead of gzipping them together... it's not really the most effective but it does work

It does not. The pipe command leaves the pipe open and successive pipes with identical strings remain open until the pipe is explicitly closed. [edit] Here's the link to the gawk documentation, but most flavors of AWK work similarly: https://www.gnu.org/software/gawk/manual/gawk.html#Close-Fil...

Wow, this is amazing. It really shows how complexity should be managed in the tool so that the user can do the naive thing and have it be accidentally optimal
Post reply on HN