Parsing Awk Is Tricky

I think you've struck on the actual reason: Rust programmers don't perceive compile times as slow, and don't really view it as a problem. Thus, nobody works on making them faster.

Every language has tradeoffs, and every language community has priorities. In general, the Rust community doesn't care about compilation speed. For now, the community has basically decided that incremental cached compilations are good enough.

Which is fair, because there's only so many engineering hours, and the language has a lot of other priorities that fast to compile languages like Go ignore.

I'm biased towards C and Go's way of thinking about language design, which I know a lot of other people hate. But, there's also the universal problem that once you introduce a feature into a language, people will have a field day using it in contexts where it's not needed. Just like Perl programmers have never met a regex they've never disliked, and C++ programming have never heard of a bad operator overload, Rust programmer's have never seen a bad procedural macro or external crate dependency. Showing just a little bit of restraint using complex or slow to compile language features goes a long way, but it seems like most devs (in all languages) can't resist. Go is partially fast to compile because it just tells devs they aren't allowed to do 90% of the slow-to-compile things that they want to do.

Powerful languages like Rust and C++ give devs the choice, and they almost always choose the slow-to-compile but elegant option. At least, until the build hits an hour, then they wish compile times were faster. For the record, I'm not bashing C++ or Rust, I'm a C++ developer by trade.

How much time does the parsing step take when compiling c++, relatively speaking? Is it actually significant compared to everything else that happens?

If you are parsing awk, you must treat any ream of whitespace that contains a newline as a visible token, which you have to reference in various places in the grammar. Your implementation will likely benefit from a switch, in the lexical analyzer, which sometimes turns off the visible newline.

I often find the missing support for slicing (like fields from 2-6, as `cut -f` can do) a handicap. I tend reach for jq instead of awk, these days.

Brian Kernighan sent Gawk maintainer Arnold Robbins an email linking to this blog post with the comment "Hindsight has a lot of benefits, it would appear."

Peter Weinberger (quoted with permission) responded:

> That's interesting, Here's some thoughts/recollections. (remember that human memory is fallible.)

> 1. Using whitespace for string concatenation, in retrospect, was probably not the ideal choice (but '+' would not have worked).

> 2. Syntax choices were in part driven by the desire for our local C programmers to find it familiar.

> 3. As creatures of a specific time and place awk shared with C the (then endearing, now irritating) property of being underspecified.

> I think that collectively we understood YACC reasonably well. We tortured the grammar until the parser came close to doing what we wanted, and then we stopped. The tools then were more primitive, but they did fit in 64K of memory.

Al Aho also replied (quoted with permission):

> Peter's observation about torturing the grammar is apt! As awk grew in its early years, the grammar evolved with it and I remember agonizing to make changes to the grammar to keep it under control (understanding and minimizing the number of yacc-generated parsing-action conflicts) as awk evolved. I found yacc's ability to point out parsing-action conflicts very helpful during awk's development. Good grammar design was very much an art in those days (maybe even today).

It's fun to hear the perspectives of the original AWK creators. I've had some correspondence with Kernighan and Weinberger before, but I think that's the first time I've been on an email thread with all three of A, W, and K.