Live data from Hacker News

Why not to use (f)lex, yacc or bison

tomassetti.me

61–70 of 91 posts

Re: Why not to use (f)lex, yacc or bison

#61

Earlier quoted context omitted.

Ruby's yacc parser is my usual go-to for telling people how not to do it. That thing's terrifying.

To be fair, ruby is not a fun language to machine-parse.

Oh absolutely. As a user, I love the ruby syntax. As a potential contributor to the codebase? NOPENOPENOPE.

Re: Why not to use (f)lex, yacc or bison

#62
I would have to check, but the resources on lex/yacc that I remember would make a point of their usability at scale. Generally it would be unlikely to wrote a faster lexer than what lex could do, but very likely to write a better parser than yacc. However writing a small lexer and/or parser _quickly_ would be easier with those tools than hand rolling one almost every time. Its great parser generators are still being developed and improved, and its fine to point out issues, but this article misses the point by a wide margin.

Re: Why not to use (f)lex, yacc or bison

#63
post #53
post #35

Earlier quoted context omitted.

Counterpoint: I used to "and I stopped reading" when I ran into grammar issues. I gradually realized that it's worth the effort to focus on the underlying ideas and evaluate that , rather than someone's 2nd language ability. People are smart, even when they talk funny, y'all. (The worth/validity of the underlying ideas is a separate matter.) Counterpoint 2: Automated grammar corrections are risky unless you're a nati…

I think it’s helpful to recall that the language is called English, not American, and the people who are experts on it think we don’t talk good.

Most English speaking people everywhere speak with some variation on "formal" English. Even the received pronunciation in South england and london is relatively modern.

If anything, some American accents are closer to the English of the late 18th century than many British ones are, and the "official" received pronunciation is not without its fair share of critics.

Re: Why not to use (f)lex, yacc or bison

#64
post #42
post #17

Is it not the case that most serious parser implementations are hand-written? In part because it makes it so much easier to provide good diagnostic messages. I feel like every other project moves from parser-generators to hand-rolled parsers and levels.

Both go and OCaml use LR parser generators, precisely because they make it easy to give excellent error messages. Basically the key insight is that you can use the parsing automaton state to classify syntax errors, which makes it as easy as falling off a log to produce good messages. The paper originating this technique is Clinton Jeffery's TOPLAS 2003 paper “Generating LR Syntax Error Messages from Examples.” This i…

Go doesn't use a generated parser. Both the parser and the lexer are hand-written:

https://github.com/golang/go/blob/master/src/go/parser/parse...

https://github.com/golang/go/blob/master/src/go/scanner/scan...

Why did you think it was generated?

Re: Why not to use (f)lex, yacc or bison

#65
post #17

Is it not the case that most serious parser implementations are hand-written? In part because it makes it so much easier to provide good diagnostic messages. I feel like every other project moves from parser-generators to hand-rolled parsers and levels.

You can check out how CockroachDB uses special error symbols with goyacc to provide good error messages for SQL passing. It’s a very interesting read really.

https://github.com/cockroachdb/cockroach/tree/master/pkg/sql...

Re: Why not to use (f)lex, yacc or bison

#66

I have used ANTLR to write a parser for a simple language and found it kind of confusing, but put it down to being new to ANTLR and to parser generators in general. The reason I chose it was for its ability to output parsers in multiple implementation languages for a single grammar - lemon et al seem to be tied to C. Are there any other systems the HN crowd would recommend for this use case?

I've been poking at coco/r recently and it seems to have a nice collection of backends.

Not sure how maintained it is, spent an afternoon getting the Python version ported over to py3 (which wasn't really all that hard) to learn how it works.

As TFA states you can't reuse grammars for multiple languages because the actions are declared inline but a few of the other complaints aren't an issue due to the way the generated parser does it thing -- quite well designed IMHO.

Re: Why not to use (f)lex, yacc or bison

#67
post #32
post #17

Is it not the case that most serious parser implementations are hand-written? In part because it makes it so much easier to provide good diagnostic messages. I feel like every other project moves from parser-generators to hand-rolled parsers and levels.

Community wisdom has long held that good error recovery is impossible for automatically generated parsers and I unquestioningly accepted that wisdom for years. When I was building a Yacc system in Rust, I eventually investigated further, and found that there's loads of work in this area over several decades. None of it ever found its way into real parsers, probably because it was too slow. I extended some of that wor…

For YACC/bison yes, but for peg parser generators not. They also don't need a lexer.

Re: Why not to use (f)lex, yacc or bison

#68
post #42

Earlier quoted context omitted.

Both go and OCaml use LR parser generators, precisely because they make it easy to give excellent error messages. Basically the key insight is that you can use the parsing automaton state to classify syntax errors, which makes it as easy as falling off a log to produce good messages. The paper originating this technique is Clinton Jeffery's TOPLAS 2003 paper “Generating LR Syntax Error Messages from Examples.” This i…

Go doesn't use a generated parser. Both the parser and the lexer are hand-written: https://github.com/golang/go/blob/master/src/go/parser/parse... https://github.com/golang/go/blob/master/src/go/scanner/scan... Why did you think it was generated?

If you look closely at the Go code you'll see it was once C code generated by some parser generater (which I assume was Bison). Believe it was Russ Cox that wrote a program that transformed the generated C code into Go.

Edit: I forked the compiler a while ago to add maybe types (a la Rust result). Looks like the compiler code has changed quite a bit from when I was playing w/ it.

Re: Why not to use (f)lex, yacc or bison

#69

Earlier quoted context omitted.

Go doesn't use a generated parser. Both the parser and the lexer are hand-written: https://github.com/golang/go/blob/master/src/go/parser/parse... https://github.com/golang/go/blob/master/src/go/scanner/scan... Why did you think it was generated?

If you look closely at the Go code you'll see it was once C code generated by some parser generater (which I assume was Bison). Believe it was Russ Cox that wrote a program that transformed the generated C code into Go. Edit: I forked the compiler a while ago to add maybe types (a la Rust result). Looks like the compiler code has changed quite a bit from when I was playing w/ it.

That's a long time ago. You can see vestiges of it here and there (mostly in deeper areas of the compiler, which goes all the way back to Plan 9), but the parser now looks very clean.

Re: Why not to use (f)lex, yacc or bison

#70
post #17

Is it not the case that most serious parser implementations are hand-written? In part because it makes it so much easier to provide good diagnostic messages. I feel like every other project moves from parser-generators to hand-rolled parsers and levels.

Guy Steele said this in the Dynamic Languages Wizards series, in the panel on language design [1]:

Be sure that your language will parse. It seems stupid to sit down and start designing constructs and not worry how they will fit together. You can get a language that's difficult if not impossible to parse, not only for a computer, but for a person. I use YACC constantly as a check of all my language designs, but I very seldom use YACC in the implementation. I use it as a tester, to be sure that it's LR(1) ... because if a language is LR(1) it's more likely that a person can deal with it.

[1] https://youtu.be/agw-wlHGi0E?t=4145

Post reply on HN