Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing Awk Is Tricky

raygard.net

Post reply on HN
Print view View original

31–40 of 95 posts

Re: Parsing Awk Is Tricky

#31

Post by forinti »

dotancohen wrote: post #21
DonHopkins wrote: post #15

Earlier quoted context omitted.

You just need to have the skill to overcome whatever non-technical, legacy, lack of education, or poor judgement issues that are steamrolling you into choosing to use awk instead of a sane rational decent modern efficient maintainable language.

Perl, then?

As a dare from a friend I compared my Perl solution to an AWK solution:

  $time perl -MData::Dumper -ne '$n{length($_)}++; END {print Dumper(%n)}' bigfile.txt

  $VAR1 = '1088';
  $VAR2 = 349647;

  real    0m1.326s
  user    0m0.814s
  sys     0m0.371s

  $time awk 'length($0) > max { max=length($0) } END { print max }' bigfile.txt

  1087

  real    0m21.400s
  user    0m18.596s
  sys     0m0.455s
I prefer Perl, but I have no issue with AWK and I actually use it frequently.

Re: Parsing Awk Is Tricky

#32

Post by mmsc »

Awk is something that I think every programmer and especially every sysadmin should learn. 8 like the comparison at the end and have never heard of nnawk or bbawk before.

I recently made a dashboard to compare four versions of awk output together, since not all awk scripts I'll run the same on each version: https://megamansec.github.io/awk-compare/ I'll have to add those:)

Re: Parsing Awk Is Tricky

#33

Post by kragen »

orwin wrote: post #26
keybored wrote: post #19

Earlier quoted context omitted.

> According to its author, the main reason D has very fast compile time (as long as you avoid the CTFE) is because of the language design decisions avoid the notorious symbols that can complicated symbol table just like happened in C++ and the popular > overloading for I/O and shifting. But the fact that Rust come much later than C++ and D but still slow to compile is bewildering to say the least. The reasons why Rus…

Rust isn't particulary slow to compile as long as you keep opt-level to 1 and the number of external library minimal. But even them it isn't as slow as C++ (but i write shit C++ code, i've heard that modern C++ is way better, i learned with C++98 and never really improved my style despite using C++11).

http://canonical.org/~kragen/sw/dev3/gcd.rs, which uses no external libraries, takes 400–450ms to compile with rustc -C opt-level=1 gcd.rs (buggy program, i know). gcc 12, which is not anyone's idea of a fast c compiler, compiles the c equivalent http://canonical.org/~kragen/sw/dev3/gcd.c in 70–90ms, so the rust compiler is 300–500% slower

tcc, which is most people's idea of a fast c compiler, compiles gcd.c in 8–9ms, so the rust compiler is 4300–5500% slower

so from my point of view 'rust isn't particularly slow to compile' is off by about an order of magnitude

is it as slow as c++? well, g++ compiles the c++ version of the same code http://canonical.org/~kragen/sw/dev3/gcd.cc in 460–490ms. so in this case compiling rust is, yeah, on the order of 10% faster than compiling c++? i feel like that's basically the same

of course you can make compiling c++ arbitrarily slow with templates

Re: Parsing Awk Is Tricky

#34

Post by kragen »

mst wrote: post #29
dotancohen wrote: post #21

Earlier quoted context omitted.

Perl, then?

The rule of thumb back at Netcraft was to prototype in awk/sed for brevity/expressiveness and then port to perl for production use for performance reasons. Been a couple decades since I was wrangling the survey systems there though, no idea what it looks like now.

i very much appreciate the server surveys; for a time i read the report every month!

Re: Parsing Awk Is Tricky

#35

Post by Chris2048 »

mmsc wrote: post #32

Awk is something that I think every programmer and especially every sysadmin should learn. 8 like the comparison at the end and have never heard of nnawk or bbawk before. I recently made a dashboard to compare four versions of awk output together, since not all awk scripts I'll run the same on each version: https://megamansec.github.io/awk-compare/ I'll have to add those:)

> every programmer and especially every sysadmin should learn

There are lots of things "every should learn", usually by people who already did so. I still have a bunch of AI/ML items on that list too.

What's the advantage of learning AWK over Perl?

Re: Parsing Awk Is Tricky

#36

Post by tgv »

thomasmg wrote: post #10
jasone wrote: post #3

Earlier quoted context omitted.

Hard disagree. Yacc has unnecessary footguns, in particular the fallout from using LALR(1), but more modern parser generators like bison provide LR(1) and IELR(1). Hand-rolled recursive descent parsers as well as parser combinators can easily obscure implicit resolution of grammar ambiguities. A good LR(1) parser generator enables a level of grammar consistency that is very difficult to achieve otherwise.

> Hand-rolled recursive descent parsers as well as parser combinators can easily obscure implicit resolution of grammar ambiguities. Could you give a concrete, real-life example of this? I have written many recursive-descent parsers and never ran into this problem (Apache Jackrabbit Oak SQL and XPath parser, H2 database engine, PointBase Micro database engine, HypersonicSQL, NewSQL, Regex parsers, GraphQL parsers, an…

The original comment says that using yacc/bison is "fundamentally misguided." But parser generators make it easy to add a correct parser to your project. It's obviously not the only way. Hand-rolling has a bunch of pitfalls, and easily leads to apparently correct behavior that does weird things on untested input. Your comment then is a bit like: I've never had memory corruption in C, so Rust/Java/etc. is for toy projects only.

Re: Parsing Awk Is Tricky

#37

Post by orwin »

kragen wrote: post #33
orwin wrote: post #26

Earlier quoted context omitted.

Rust isn't particulary slow to compile as long as you keep opt-level to 1 and the number of external library minimal. But even them it isn't as slow as C++ (but i write shit C++ code, i've heard that modern C++ is way better, i learned with C++98 and never really improved my style despite using C++11).

http://canonical.org/~kragen/sw/dev3/gcd.rs , which uses no external libraries, takes 400–450ms to compile with rustc -C opt-level=1 gcd.rs (buggy program, i know). gcc 12, which is not anyone's idea of a fast c compiler, compiles the c equivalent http://canonical.org/~kragen/sw/dev3/gcd.c in 70–90ms, so the rust compiler is 300–500% slower tcc, which is most people's idea of a fast c compiler, compiles gcd.c in 8–9m…

> of course you can make compiling c++ arbitrarily slow with templates

This might be my problem :/ (template are the closest to metaprogramming I can find outside of Lisps)

Tbf I was mostly comparing my experience with Rust, SBCL and C++, to me it was a given that C was an order of magnitude faster (3 order of magnitude seems a bit much). I found opt-level=1 quite early and managed to feel way better about rust and let C++ go (i was toying with polynomial regressions) (I rolled my own matrix library :D never do that!)

Thank you for the informations.

Re: Parsing Awk Is Tricky

#38

Post by kragen »

orwin wrote: post #37
kragen wrote: post #33

Earlier quoted context omitted.

http://canonical.org/~kragen/sw/dev3/gcd.rs , which uses no external libraries, takes 400–450ms to compile with rustc -C opt-level=1 gcd.rs (buggy program, i know). gcc 12, which is not anyone's idea of a fast c compiler, compiles the c equivalent http://canonical.org/~kragen/sw/dev3/gcd.c in 70–90ms, so the rust compiler is 300–500% slower tcc, which is most people's idea of a fast c compiler, compiles gcd.c in 8–9m…

> of course you can make compiling c++ arbitrarily slow with templates This might be my problem :/ (template are the closest to metaprogramming I can find outside of Lisps) Tbf I was mostly comparing my experience with Rust, SBCL and C++, to me it was a given that C was an order of magnitude faster (3 order of magnitude seems a bit much). I found opt-level=1 quite early and managed to feel way better about rust and l…

yeah! you can get an enormous amount of metaprogramming mileage out of c++ templates. i think the pattern-matching paradigm embodied by sfinae is maybe a better fit for, effectively, user-defined language extensions, than the more straightforward imperative approach lisp uses by default. but c++ templates are unnecessarily hard to debug i think, for reasons that aren't inherent to the pattern-matching paradigm

i didn't get c to compile three orders of magnitude faster, just 44×–56× faster (4400% to 5500%). sorry to be confusing!

i've certainly experienced the temptation to roll my own matrix library more than once, and i'll definitely have to do it at least once for the zorzpad. i may do something this week in order to understand the simplex method better; my operations research class was years ago, and i've forgotten how it works, probably because i never implemented it in software, just worked through examples by hand and on the chalkboard

Re: Parsing Awk Is Tricky

#39

Post by actuallyalys »

tannhaeuser wrote: post #9
v3ss0n wrote: post #5

Reading awk as a human is hard too. And performance of awk is crap. A lot slower than most interpreter language out there. I had replaced all the awk scripts in python and everything is a lot faster.

Discussing performance only makes sense in the context of a particular awk implementation, like TFA is doing as well. If you‘re (stuck) on gawk, try setting LANG=C to prevent Unicode/multi-byte regexp execution, or switch to mawk (which according to [1] is much faster than cpython). [1]: https://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-...

Honestly only makes sense in the context of a Python library and implementation as well, since so many libraries use C extensions in order to speed up processing. Also, Python has gotten a lot faster over time.

Re: Parsing Awk Is Tricky

#40

Post by rlonstein »

Chris2048 wrote: post #35
mmsc wrote: post #32

Awk is something that I think every programmer and especially every sysadmin should learn. 8 like the comparison at the end and have never heard of nnawk or bbawk before. I recently made a dashboard to compare four versions of awk output together, since not all awk scripts I'll run the same on each version: https://megamansec.github.io/awk-compare/ I'll have to add those:)

> every programmer and especially every sysadmin should learn There are lots of things "every should learn", usually by people who already did so. I still have a bunch of AI/ML items on that list too. What's the advantage of learning AWK over Perl?

- Awk is defined in POSIX

- Awk is on more systems than Perl

- Awk has more implementations than Perl

Post reply on HN