Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing Awk Is Tricky

raygard.net

Post reply on HN
Print view View original

81–90 of 95 posts

Re: Parsing Awk Is Tricky

#81

Post by 1vuio0pswjnm7 »

benhoyt wrote: post #79

Brian Kernighan sent Gawk maintainer Arnold Robbins an email linking to this blog post with the comment "Hindsight has a lot of benefits, it would appear." Peter Weinberger (quoted with permission) responded: > That's interesting, Here's some thoughts/recollections. (remember that human memory is fallible.) > 1. Using whitespace for string concatenation, in retrospect, was probably not the ideal choice (but '+' would…

"The tools were more primitive, but they did fit in 64k of memory."

I will take "primitive" over present-day bloat and complexity every time, quirks and all.

That programs fitting in 64K of memory have remained in continuous use and the subject of imitation for so long must be a point of pride for the authors. From what I have seen, contemporary software authors are unlikely to ever achieve such longevity.

Re: Parsing Awk Is Tricky

#82

Post by HeroicKatora »

kragen wrote: post #33
orwin wrote: post #26

Earlier quoted context omitted.

Rust isn't particulary slow to compile as long as you keep opt-level to 1 and the number of external library minimal. But even them it isn't as slow as C++ (but i write shit C++ code, i've heard that modern C++ is way better, i learned with C++98 and never really improved my style despite using C++11).

http://canonical.org/~kragen/sw/dev3/gcd.rs , which uses no external libraries, takes 400–450ms to compile with rustc -C opt-level=1 gcd.rs (buggy program, i know). gcc 12, which is not anyone's idea of a fast c compiler, compiles the c equivalent http://canonical.org/~kragen/sw/dev3/gcd.c in 70–90ms, so the rust compiler is 300–500% slower tcc, which is most people's idea of a fast c compiler, compiles gcd.c in 8–9m…

I couldn't quite replicate those numbers (rustc 1.78, gcc 14, g++ 14) with a recent state. On my machine (Ryzen 9 7900X, LVM on NVMe) it's rustc 60-80ms, gcc 20-30ms and tcc in 2ms. Intererestingly, g++ is still 200ms on that machine. Activating time and the builtin time-passes in rustc here's also an interesting observation: rustc spends 47ms of its time in sys and 23ms in user compared to I think the test is uultimately non-sensical for the question being posed here. It doesn't reveal anything insightful about scaling to real world program sizes, either. The time of rustc is dominated by the platform linker anyways. Sure, one might argue that this points out Rust as relying too much on the linker and creating too many unused symbols. But the question of whether this is caused by the language and in particular its syntactical choices .. should at that point be answered with probably not. It's not a benchmark you want to compare by percentage speedups anyways since it's probably dominated by constant time costs for any of the batteries included standard library languages compared to C.

Re: Parsing Awk Is Tricky

#83

Post by kragen »

HeroicKatora wrote: post #82
kragen wrote: post #33

Earlier quoted context omitted.

http://canonical.org/~kragen/sw/dev3/gcd.rs , which uses no external libraries, takes 400–450ms to compile with rustc -C opt-level=1 gcd.rs (buggy program, i know). gcc 12, which is not anyone's idea of a fast c compiler, compiles the c equivalent http://canonical.org/~kragen/sw/dev3/gcd.c in 70–90ms, so the rust compiler is 300–500% slower tcc, which is most people's idea of a fast c compiler, compiles gcd.c in 8–9m…

I couldn't quite replicate those numbers (rustc 1.78, gcc 14, g++ 14) with a recent state. On my machine (Ryzen 9 7900X, LVM on NVMe) it's rustc 60-80ms, gcc 20-30ms and tcc in 2ms. Intererestingly, g++ is still 200ms on that machine. Activating time and the builtin time-passes in rustc here's also an interesting observation: rustc spends 47ms of its time in sys and 23ms in user compared to I think the test is uultim…

thank you very much for the failed replication!

it's interesting, my machine is fairly similar—ryzen 5 3500u, rustc 1.63.0, luks on nvme. is it possible that rustc has gotten much faster since 1.63?

while i agree that it's not the most important test for day-to-day use, i don't agree that it falls to the level of nonsensical. how fast things are determines how you can use them. tcc and old versions of gcc are fast enough that you could very reasonably generate a c file, compile it into a new shared object, dlopen it, and call it, every screen frame. there are some languages, like gforth, that actually implement their ffi in such a way, and sitkack and i have both done some experiments with inline c and jit compilation by this mechanism

i do agree that the syntactical choices of the language have relatively little to do with it, and your rustc measurements provide strong evidence of that—though perhaps it is somewhat unfavorable for c++ that it commonly has to tokenize many megabytes of header files and do the moral equivalent of text replacement to implement parametric polymorphism

Re: Parsing Awk Is Tricky

#84

Post by emmelaich »

rlonstein wrote: post #40
Chris2048 wrote: post #35

Earlier quoted context omitted.

> every programmer and especially every sysadmin should learn There are lots of things "every should learn", usually by people who already did so. I still have a bunch of AI/ML items on that list too. What's the advantage of learning AWK over Perl?

- Awk is defined in POSIX - Awk is on more systems than Perl - Awk has more implementations than Perl

POSIX not really relevant, more systems? Debatable. More implementations could be seen as a negative.

Perl is more regular than Awk for the simple cases and is more usable for anything that isn't merely iterating over input.

Of course, you shouldn't any of awk/perl/shell for tasks that aren't being run by you or are over say 20 lines long.

Re: Parsing Awk Is Tricky

#85

Post by kazinator »

Chris2048 wrote: post #68
rlonstein wrote: post #65

Earlier quoted context omitted.

Whatever you think my opinion is of Perl you're probably wrong and the tone of your advocacy is kind of odd. Awk is older and as a part of POSIX the version found on unix-like environments will be (outside of extensions) compatible with others. If one or one without the extensions you want isn't present you can choose an implementation, even one in Go and it'll work. Perl, and I've been writing Perl since Perl4, does…

The first and last items in your list provide no reason why they are relevant, there is no "tone", nor "advocacy" - it's not "odd" to ask for that context, as given here.

Awk is found in small-ish embedded systems that don't have no reason to waste space on Perl or anything like it.

One reason for this is that the popular BusyBox project includes an Awk implementation: BusyBox Awk.

Pretty much everywhere there is BusyBox, there is an Awk, unless someone went out of their way to compile it out of the BuxyBox binary.

Re: Parsing Awk Is Tricky

#86

Post by [deleted] »

Chris2048 wrote: post #61
mbivert wrote: post #60

Earlier quoted context omitted.

> What's the advantage of learning AWK over Perl? Getting awk in your head (fully) takes about an afternoon: reading the (small and exhaustive) man page, going through a few examples, trying to build a few toys with it. Perl requires much, much more effort. Great gain/investment ratio.

another commenter said something similar - But nothing says you have to learn everything - you can learn a subset of perl that does everything you would want to do (with awk), would that take as long?

[deleted]

Re: Parsing Awk Is Tricky

#87

Post by arp242 »

stouset wrote: post #69
Chris2048 wrote: post #64

Earlier quoted context omitted.

but, then couldn't you use "cut" as even simpler syntax?

`cut` doesn’t work natively on data that’s been aligned with multiple spaces, you need a `tr -s` pass first. It also doesn’t let you reorder or splice together fields. I used it for years but now that I have a working understanding of `awk` I have never looked back.

FreeBSD cut has -w for that ("split by any amount of whitespace"), but that never made it into GNU cut. Sad, because it's mega useful.

Of course awk can do much more, but if all you want is "| awk '{print $2}'" then "cut -wf2" is so much more convenient.

Re: Parsing Awk Is Tricky

#88

Post by v3ss0n »

mananaysiempre wrote: post #22
v3ss0n wrote: post #5

Reading awk as a human is hard too. And performance of awk is crap. A lot slower than most interpreter language out there. I had replaced all the awk scripts in python and everything is a lot faster.

> And performance of awk is crap. [...] I had replaced all the awk scripts in python and everything is a lot faster. My experience points exactly the other way: for data-processing tasks, especially streaming ones, even Gawk is a lot faster than Python (pre-3.11), and apparently I’m not the only one[1]. If you’re not satisfied with Gawk’s performance, though, try Nawk[2] or, even better, Mawk[3]. (And stick to POSIX…

You gotta share the code how you are doing it. If you are using awk alternative, you would be comparing against pandas or pypy. I will do a comparison as soon as I am free.

Re: Parsing Awk Is Tricky

#89

Post by v3ss0n »

tannhaeuser wrote: post #9
v3ss0n wrote: post #5

Reading awk as a human is hard too. And performance of awk is crap. A lot slower than most interpreter language out there. I had replaced all the awk scripts in python and everything is a lot faster.

Discussing performance only makes sense in the context of a particular awk implementation, like TFA is doing as well. If you‘re (stuck) on gawk, try setting LANG=C to prevent Unicode/multi-byte regexp execution, or switch to mawk (which according to [1] is much faster than cpython). [1]: https://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-...

We gotta compare against pypy,or cpython plus pandas then

Re: Parsing Awk Is Tricky

#90

Post by HeroicKatora »

kragen wrote: post #83
HeroicKatora wrote: post #82

Earlier quoted context omitted.

I couldn't quite replicate those numbers (rustc 1.78, gcc 14, g++ 14) with a recent state. On my machine (Ryzen 9 7900X, LVM on NVMe) it's rustc 60-80ms, gcc 20-30ms and tcc in 2ms. Intererestingly, g++ is still 200ms on that machine. Activating time and the builtin time-passes in rustc here's also an interesting observation: rustc spends 47ms of its time in sys and 23ms in user compared to I think the test is uultim…

thank you very much for the failed replication! it's interesting, my machine is fairly similar—ryzen 5 3500u, rustc 1.63.0, luks on nvme. is it possible that rustc has gotten much faster since 1.63? while i agree that it's not the most important test for day-to-day use, i don't agree that it falls to the level of nonsensical . how fast things are determines how you can use them. tcc and old versions of gcc are fast e…

Thank you for re-validating the numbers on your end, it's indeed very possible. There's been quite a few improvements in those versions. Though the effect size does not quite fit with most of the optimizations I can recall, maybe it's much more related to optimizations to the standard library's size and linking behavior.

With regards to standard use, for many users the scenario is definitely not common. I'd rather rustc be an effective screw driver and a separate hammer be built than try to mangle both into the same tool. By that I mean, it's very clear which portion of the compiler must be repurposed here. The hard question is whether the architecture is amenable to alternative linker backends that serve your use-case. I'm afraid I can't answer that conclusively. Only so much, the conceptual conflict of Rust is that linkining is a very memory-safety critical part of the process. And with its compilation module model it relinks everything into the resulting binary / library which includes a large std and dependency tree even if much of this is removed by the step. Maybe that can be changed; and relying a tool whose interface was ultimately designed with C in mind is also far from optimal to compute those outputs and inputs. It's hard to say how much of it stems from compatibility concerns and compatibility overheads and how much is fundamental to the language's design which could be shed for a pure build process.

With regards to C++, I suspect it's rooted in the fact that parsing it requires in principle the implementation of a complete consteval engine. The language has a dependency loop between parsing and codegen. This of course, is not how data should be laid out for executing fast programs on it. It's quite concerning given the specifications still contains the bold faced lie that "The disambiguation is purely syntactic" (6.8; 1) for typenames vs non-typenames to parse constructors from declarations which at the present can require arbitrary template specialization. It might be interesting to see if those two headers in your example already execute some of these dependency loops but it's hard for me to think of an experiment to validate any of this. Maybe you have ideas, is there something like time-passes?

Post reply on HN