Live data from Hacker News

Awk: The Power and Promise of a 40-Year-Old Language

fosslife.org

101–110 of 122 posts

Re: Awk: The Power and Promise of a 40-Year-Old Language

#101
post #91

Earlier quoted context omitted.

But perl was already the "standard" for other system/config utilities, no?

I don't know what we mean by "standard," but I found a number of perl references with the following shell fragment: $ for x in $(echo $PATH|sed 's/:/ /g'); do file $x/*|grep perl;done All but two hits were in /usr/sbin, and /usr/bin. I isolated those files with: $ file /usr/sbin/* | awk '/perl/{sub(/:.*/,"");sub(/^.*[/]/,"");printf "%s, ", $0}';echo '' The sbin results are: adduser, fw_update, pkg_add, pkg_check, pkg…

> A perl script can't pledge() or unveil()

It doesn't seem to support all of OpenBSD's privilege separation, but there are OpenBSD::Unveil(3p), OpenBSD::Pledge(3p), and https://github.com/rfarr/Unix-Pledge

https://bronevichok.ru/posts/pledge.html

Re: Awk: The Power and Promise of a 40-Year-Old Language

#102

Earlier quoted context omitted.

The biggest reason to learn AWK, IMO, is that it's on pretty much every single linux distribution. You might not have perl or python. You WILL have AWK. Only the most minimal of minimal linux systems will exclude it. Even busybox includes awk. That's how essential it's viewed.

I'm curious what linux distros don't have either some version of perl or python. I like awk, mind, but this is not necessarily (IME) a good argument for it.

The better question might be "which Linux distro's don't have perl or python installed by default" as a lot of people are working on systems where they can't just add additional packages.

Perl has been getting cut from minimal builds of distro's for a while. Default installed version of python is a bit of a crap-shoot, nevermind which modules you might happen to have available.

Re: Awk: The Power and Promise of a 40-Year-Old Language

#103
post #82
post #2

HN discussion threads for some of the links mentioned in the article: * Using AWK and R to parse 25TB - https://news.ycombinator.com/item?id=20293579 * Command-line Tools can be 235x Faster than a Hadoop Cluster - https://news.ycombinator.com/item?id=17135841 * The State of the AWK - https://news.ycombinator.com/item?id=23240800 For awk alternative implementations, I'm keeping an eye on frawk [0]. Aims to be faster,…

CSV is a complicated format but that does not mean awk is incapable of dealing with it. https://www.gnu.org/software/gawk/manual/html_node/Splitting... https://github.com/e36freak/awk-libs/blob/master/csv.awk https://raw.githubusercontent.com/Nomarian/Awk-Batteries/mas...

> CSV is a complicated format

Surprisingly and unnecessarily so:

> ["DSV"] is to Unix what CSV (comma-separated value) format is under Microsoft Windows and elsewhere outside the Unix world. CSV (fields separated by commas, double quotes used to escape commas, no continuation lines) is rarely found under Unix.

> In fact, the Microsoft version of CSV is a textbook example of how not to design a textual file format. Its problems begin with the case in which the separator character (in this case, a comma) is found inside a field. The Unix way would be to simply escape the separator with a backslash, and have a double escape represent a literal backslash. This design gives us a single special case (the escape character) to check for when parsing the file, and only a single action when the escape is found (treat the following character as a literal). The latter conveniently not only handles the separator character, but gives us a way to handle the escape character and newlines for free. CSV, on the other hand, encloses the entire field in double quotes if it contains the separator. If the field contains double quotes, it must also be enclosed in double quotes, and the individual double quotes in the field must themselves be repeated twice to indicate that they don't end the field.

> The bad results of proliferating special cases are twofold. First, the complexity of the parser (and its vulnerability to bugs) is increased. Second, because the format rules are complex and underspecified, different implementations diverge in their handling of edge cases. Sometimes continuation lines are supported, by starting the last field of the line with an unterminated double quote — but only in some products! Microsoft has incompatible versions of CSV files between its own applications, and in some cases between different versions of the same application (Excel being the obvious example here).

The Art of Unix Programming http://www.catb.org/~esr/writings/taoup/html/ch05s02.html

Re: Awk: The Power and Promise of a 40-Year-Old Language

#104
post #94

Earlier quoted context omitted.

A nice thing about awk vs. Perl/Python: there's a small focused set of things to learn. Once you learn them you're done. This suggests an opening for a Perl/Python intro focused on the exact same tasks, admittedly. That seems more realistic for Perl -- unless there's someone who writes Python one-liners at the shell?

I don't think true python "one liners" are a thing, but the awkward thing about awk is sits in this place where what you are doing is complicated enough you need awk, but simple enough you need a one liner? Those cases have been exceedingly few and far between for me enough that every time I want to reach for awk I have to go lookup how to do anything more complex than printing fields. That completely defeats the poi…

I think that is a good point, that often writing a short python script is usually the best solution.

I use awk (and python) daily at work. I work with a lot of flat files, and I use awk when I am doing data quality checks. One of the "sweet spots" it hits for me is when I need to group data by value, or other relatively simple aggregations.

Re: Awk: The Power and Promise of a 40-Year-Old Language

#105
post #13

"A good programmer uses the most powerful tool to do a job. A great programmer uses the least powerful tool that does the job." I believe this, and I always try to find the combination of simple and lightweight tools which does the job at hand correctly. Awk sometimes proves surprisingly powerful. Just look at the concision of this awk one liner doing a fairly complex job: zcat large.log.gz | awk '{print $0 | "gzip -…

I really love that quote "..A good programmer...", do you have a source?

Re: Awk: The Power and Promise of a 40-Year-Old Language

#106
post #29
post #4

Gawk's ability to extend it with C code is interesting as well, and pretty straightforward. Here's the source for the fork() extension that ships with gawk...it's ~150 lines or so: https://git.savannah.gnu.org/cgit/gawk.git/tree/extension/fo... I was able to make a (terrible/joke/but-it-kinda-works) web server with gawk using the extensions that ship with it: https://gist.github.com/willurd/5720255#gistcomment-314300…

My opinion that belongs to me is as follows. This is how it goes. The next thing I'm going to say is my opinion. The C interop and name-spaces (also in gawk) is a bridge too far for me. By the time you need one of those, it's time to look for another language. Awk is just not enough of a language to write serious programs in. And I really like awk. It has enabled great scripting not only for log files, but also for d…

It's good you're unapologetic. At the same time, these sort of features are what I love as they avoid me having to move onwards to something new, and start near ground zero. Living by the mantra "Do 2 things 1000 times, not 1000 things 2 times."

Re: Awk: The Power and Promise of a 40-Year-Old Language

#108
post #91

Earlier quoted context omitted.

I don't know what we mean by "standard," but I found a number of perl references with the following shell fragment: $ for x in $(echo $PATH|sed 's/:/ /g'); do file $x/*|grep perl;done All but two hits were in /usr/sbin, and /usr/bin. I isolated those files with: $ file /usr/sbin/* | awk '/perl/{sub(/:.*/,"");sub(/^.*[/]/,"");printf "%s, ", $0}';echo '' The sbin results are: adduser, fw_update, pkg_add, pkg_check, pkg…

> A perl script can't pledge() or unveil() It doesn't seem to support all of OpenBSD's privilege separation, but there are OpenBSD::Unveil(3p), OpenBSD::Pledge(3p), and https://github.com/rfarr/Unix-Pledge https://bronevichok.ru/posts/pledge.html

Did not know that, thanks.

Re: Awk: The Power and Promise of a 40-Year-Old Language

#109
post #14

> Very few people still code with the legacies of the 1970s: ML, Pascal, Scheme, Smalltalk. Arguably, the software world would be better off if more people did code with those 1970s languages, than with the ones we are stuck with now. And that applies to Awk, too. As the author quotes Neil Ormos stating, Awk is well suited for personal computing , something which we have gotten further and further from at the same ti…

Oh man, you sound like a long lost friend. As someone who struggles to adopt really anything post ~1995 in the programming world, I couldn't agree more. I've worked for Fortune 100s my whole career; mostly in big data problem-spaces, before it ever was cool (if it even is now?), and I really feel all the problems people perceive today were solved all the way back to the 1960s (i.e. Snobol4). I understand for modern web and mobile contexts, sure there is new fancy tools for that; but as you said, in the personal computing space, the proper tools have existed for decades.

Re: Awk: The Power and Promise of a 40-Year-Old Language

#110
When you have a standardized problem setting like the implicit loop in awk, n alternative to a whole new programming language is a simple This design lets you retain easy access to large sets of pre-existing libraries as well as have a "compiled/statically typed" situation, if you want. It also leverages familiarity with your existing programming languages. I adapted a similar small program like this to emit a C program, but anything else is obviously pretty easy. Easy is good. Familiar is good.

Interactivity-wise, with a TinyC/tcc fast running compiler backend my `rp` programs run sub-second from ENTER to completion on small data. Even with not optimizing tcc, they they still run faster than byte-compiled/VM interpreted mawk/gawk on a per input-byte basis. If you take the time to do an optimized build with gcc -O3/etc., they can run much faster.

And I leave the source code around if you want to just use the program generator as a way to save keystrokes/get a fast start on a row processing program.

Anyway, I'm not trying to start a language holy war, but just exhibit how if you rotate the problem (or your head looking at the problem) ever so slightly another answer exists in this space and is quite easy. :-)

[1] https://github.com/c-blake/cligen/blob/master/examples/rp.ni...

Post reply on HN