Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing Awk Is Tricky

raygard.net

Post reply on HN
Print view View original

91–95 of 95 posts

Re: Parsing Awk Is Tricky

#91

Post by thomasmg »

tgv wrote: post #36
thomasmg wrote: post #10

Earlier quoted context omitted.

> Hand-rolled recursive descent parsers as well as parser combinators can easily obscure implicit resolution of grammar ambiguities. Could you give a concrete, real-life example of this? I have written many recursive-descent parsers and never ran into this problem (Apache Jackrabbit Oak SQL and XPath parser, H2 database engine, PointBase Micro database engine, HypersonicSQL, NewSQL, Regex parsers, GraphQL parsers, an…

The original comment says that using yacc/bison is "fundamentally misguided." But parser generators make it easy to add a correct parser to your project. It's obviously not the only way. Hand-rolling has a bunch of pitfalls, and easily leads to apparently correct behavior that does weird things on untested input. Your comment then is a bit like: I've never had memory corruption in C, so Rust/Java/etc. is for toy proj…

> Hand-rolling has a bunch of pitfalls

I'm arguing that this is not the case in reality, and asked for concrete examples... So again I ask for a concrete example... For memory corruption, there are plenty of examples.

For parsing, I know one example that lead to problems. Interestingly, it was about using a state machine that was then modified (manually) and the result was broken. Here I argue that using a handwritten parser, instead of a state machine that is then manually modified, would not have resulted in this problem. Also, there was no randomized testing / fuzz testing, which is also a problem. This issue is still open: https://issues.apache.org/jira/browse/OAK-5367

Re: Parsing Awk Is Tricky

#92

Post by kragen »

HeroicKatora wrote: post #90
kragen wrote: post #83

Earlier quoted context omitted.

thank you very much for the failed replication! it's interesting, my machine is fairly similar—ryzen 5 3500u, rustc 1.63.0, luks on nvme. is it possible that rustc has gotten much faster since 1.63? while i agree that it's not the most important test for day-to-day use, i don't agree that it falls to the level of nonsensical . how fast things are determines how you can use them. tcc and old versions of gcc are fast e…

Thank you for re-validating the numbers on your end, it's indeed very possible. There's been quite a few improvements in those versions. Though the effect size does not quite fit with most of the optimizations I can recall, maybe it's much more related to optimizations to the standard library's size and linking behavior. With regards to standard use, for many users the scenario is definitely not common. I'd rather ru…

dunno. with respect to c++, you could probably hack together a c++ compiler setup that was more aggressive about using precompiled-header-like things. and if you're trying to abuse g++ as a jit, you could maybe write a small, self-contained header that the compiler can handle quickly, and not call any standard library functions from the generated code

Re: Parsing Awk Is Tricky

#94

Post by stouset »

arp242 wrote: post #87
stouset wrote: post #69

Earlier quoted context omitted.

`cut` doesn’t work natively on data that’s been aligned with multiple spaces, you need a `tr -s` pass first. It also doesn’t let you reorder or splice together fields. I used it for years but now that I have a working understanding of `awk` I have never looked back.

FreeBSD cut has -w for that ("split by any amount of whitespace"), but that never made it into GNU cut. Sad, because it's mega useful. Of course awk can do much more, but if all you want is "| awk '{print $2}'" then "cut -wf2" is so much more convenient.

Reordering and splicing are common enough that it’s easier just to always use awk, since the cost of rewriting one to the other is significantly higher.

Re: Parsing Awk Is Tricky

#95

Post by tgv »

thomasmg wrote: post #91
tgv wrote: post #36

Earlier quoted context omitted.

The original comment says that using yacc/bison is "fundamentally misguided." But parser generators make it easy to add a correct parser to your project. It's obviously not the only way. Hand-rolling has a bunch of pitfalls, and easily leads to apparently correct behavior that does weird things on untested input. Your comment then is a bit like: I've never had memory corruption in C, so Rust/Java/etc. is for toy proj…

> Hand-rolling has a bunch of pitfalls I'm arguing that this is not the case in reality, and asked for concrete examples... So again I ask for a concrete example... For memory corruption, there are plenty of examples. For parsing, I know one example that lead to problems. Interestingly, it was about using a state machine that was then modified (manually) and the result was broken. Here I argue that using a handwritte…

There's no reason for concrete examples, because the point was about the fundamental misguidedness of parser generators, not about problems with individual parser generators or the nice things you can do in a hand-rolled one, but to accommodate you, ANTLR gives one on its home page: "... At Twitter, we use it exclusively for query parsing in Twitter search... Samuel Luckenbill, Senior Manager of Search Infrastructure, Twitter, inc."

Also, regexps are used very often in production, and that's definitely a parser-generator of sorts.

The memory corruption example was an analog, but to spell it out: it's easier and faster to write a correct parser using flex/bison than by hand, especially for more complex languages. Parser-generators have their use, and are not fundamentally misguided. That you might want to write your own parser in some cases does not diminish that (nor vice versa).

Post reply on HN