Live data from Hacker News

Why not to use (f)lex, yacc or bison

tomassetti.me

21–30 of 91 posts

Re: Why not to use (f)lex, yacc or bison

#21
That ANTLR example with the sum does little to convince me that the code should be separate from the grammar. It being separate means that you need to look at the grammar to know what tokens you have access to anyway.

(And, really, returning something of type “Any”?)

Re: Why not to use (f)lex, yacc or bison

#22
post #17

Is it not the case that most serious parser implementations are hand-written? In part because it makes it so much easier to provide good diagnostic messages. I feel like every other project moves from parser-generators to hand-rolled parsers and levels.

For general-purpose languages, yes, but there are some exceptions. Ruby uses yacc, and Python uses a custom LL(1)-ish parser generator, which is in the process of being replaced by a custom PEG parser generator [0].

[0] https://github.com/gvanrossum/pegen

Re: Why not to use (f)lex, yacc or bison

#23
post #17

Is it not the case that most serious parser implementations are hand-written? In part because it makes it so much easier to provide good diagnostic messages. I feel like every other project moves from parser-generators to hand-rolled parsers and levels.

This is actually pretty true based upon what I've personally observed too. While Apache Spark [0] and Presto [1] use ANTLR to implement their parsers, when I searched about TypeScript's parser and noticed that they implement their own parser [2] (in TypeScript itself), the reason I was able to find was that the nuanced error messages which a language like TypeScript has to provide is only feasible by hand-writing the parser.

[0] https://github.com/apache/spark/blob/master/sql/catalyst/src...

[1] https://github.com/prestodb/presto/blob/master/presto-parser...

[2] https://github.com/microsoft/TypeScript/blob/master/src/comp...

Re: Why not to use (f)lex, yacc or bison

#25
post #17

Is it not the case that most serious parser implementations are hand-written? In part because it makes it so much easier to provide good diagnostic messages. I feel like every other project moves from parser-generators to hand-rolled parsers and levels.

This is actually pretty true based upon what I've personally observed too. While Apache Spark [0] and Presto [1] use ANTLR to implement their parsers, when I searched about TypeScript's parser and noticed that they implement their own parser [2] (in TypeScript itself), the reason I was able to find was that the nuanced error messages which a language like TypeScript has to provide is only feasible by hand-writing the…

> the nuanced error messages which a language like TypeScript has to provide is only feasible by hand-writing the parser.

I wonder if this is relevant to how bad the error messages generally are in Typescript.

Re: Why not to use (f)lex, yacc or bison

#26
post #8

Maybe I was holding it wrong but in my experience ANTLR’s performance (on JVM) was abysmal.

Do you mean the performance of ANTLR, or the performance of ANTLR-generated parsers?

I mean ANTLR-generated parsers. Apologies for the ambiguity.

Re: Why not to use (f)lex, yacc or bison

#27
post #17

Is it not the case that most serious parser implementations are hand-written? In part because it makes it so much easier to provide good diagnostic messages. I feel like every other project moves from parser-generators to hand-rolled parsers and levels.

Also note the amazing architecture behind the Golang template parser/lexer A talk by Rob Like: https://talks.golang.org/2011/lex.slide

Re: Why not to use (f)lex, yacc or bison

#28

Maybe I was holding it wrong but in my experience ANTLR’s performance (on JVM) was abysmal.

Same with Python, the performance was horrifying. Didn't seem usable for production to me, but seems better suited for prototyping a language without worrying about the parser until later.

Re: Why not to use (f)lex, yacc or bison

#29
post #22
post #17

Is it not the case that most serious parser implementations are hand-written? In part because it makes it so much easier to provide good diagnostic messages. I feel like every other project moves from parser-generators to hand-rolled parsers and levels.

For general-purpose languages, yes, but there are some exceptions. Ruby uses yacc, and Python uses a custom LL(1)-ish parser generator, which is in the process of being replaced by a custom PEG parser generator [0]. [0] https://github.com/gvanrossum/pegen

Ruby's yacc parser is my usual go-to for telling people how not to do it. That thing's terrifying.

Re: Why not to use (f)lex, yacc or bison

#30
post #4

Last I checked, the interface between lex and yacc (resp flex and bison) is by mutating static global variables. Is that still the case? It always struck me as exceptionally bad design.

I don't think you are particularly restricted to doing this, but lex/yacc was conceived and designed at a time (1970s) when memory was limited, and speed and size of code that could be parsed was valued much higher than structured code and being able to provide good error messages. You simply can't do a recursive descent parser with an in memory AST in 64KiB of RAM while being able to parse any meaningful size source file. You had to serialize the problem and output a sequential AST/intermediate opcodes on the fly. FSA based lexers and parsers are beautiful in how little RAM they actually require and how fast they are, albeit clunky and very low level by modern standards.
Post reply on HN