What I don't like about any of these parsers, including ANTLR is that for real languages you get not a clear EBNF grammar, but terrible mix of declarative and imperative statements. I was pretty sure PEG is the modern way to go, but looks like it's not. So, what is then?
Why not to use (f)lex, yacc or bison
81–90 of 91 posts
Re: Why not to use (f)lex, yacc or bison
#82This whole article is very polemic and promotes an ANTLR book. It is futile to comment in detail because nearly every statement is false.
..wow, this is a bit strong. I would encourage you to be more respectful of persons who work in this very specific field and share their ideas. There are always persons behind some work you insult so easily. You know, there are over 2M persons who read our articles. A few hundred also bought a book or a video-course from us, but the vast majority just got some information from free, and we like it in this way. I do n…
It is irrelevant how many people read your articles; the critics will also show up as readers in the statistics.
Let's start the professional experience with another person's view:
https://research.swtch.com/yyerror
"Seibel: And are there development tools that just make you happy to program?
Thompson: I love yacc. I just love yacc. It just does exactly what you want done. Its complement, lex, is horrible. It does nothing you want done.
Seibel: Do you use it anyway or do you write your lexers by hand?
Thompson: I write my lexers by hand. Much easier."
I happen to like both bison and flex, which are relatively easy to use and bug-free in my experience. Yet your article spreads hundreds of lines of FUD about these tools, a strategy that many ANTLR people use.
I have used ANTLR. It is not intuitive, the documentation is horrible, if you happen to find some advice on Stackoverflow it is likely to be for another one of the incompatible versions.
I suppose if you use ANTLR long enough, these problems go away. But bison or Menhir don't have these problems in the first place.
Re: Why not to use (f)lex, yacc or bison
#83Is it not the case that most serious parser implementations are hand-written? In part because it makes it so much easier to provide good diagnostic messages. I feel like every other project moves from parser-generators to hand-rolled parsers and levels.
The reasoning is pretty simple. Writing a parser for a sane language won't take significantly more than a month for your average developer. If you have a whole team working on the same compiler for decades then the pay off becomes pretty obvious.
One of the subtle things hidden in your statement (which I do somewhat agree with) is “sane language”.
One of the more memorable parsers I’ve worked on was a parser for SPICE netlists. I started out believing that it wasn’t going to be too big of a deal, and ended up sinking a ton of time into it. Ultimately the company (as far as I know) ended up just buying a $40k license to some obscure company that had one, because there was a constant cat-and-mouse game of getting it working right and then discovering a customer who did even more strange stuff that was somehow accepted by the 3rd-party SPICE sim, but wouldn’t be accepted by ours.
I’m assuming that PHP has evolved a ton since I encountered this, but IIRC at one point you couldn’t do `$foo[$baz]($zap)` to call an anonymous function stored in an array, rather you had to `$t = $foo[$baz]; $t($zap)`. At the time (young and naive), I just couldn’t comprehend how a sane parser wouldn’t parse the first form, and then I started looking at how it was implemented...
Re: Why not to use (f)lex, yacc or bison
#84"ANTLR instead is more actively developed. It has been re-written from scratch a few times during the years, so the code has is of good quality." ... rewrites improve code quality ... right ... right???
I first learned Antlr way back about 20 years ago and found it fairly impressive. So when I go to write a new parser or lexer I always go take a look again... But every time I take a new look at Antlr there's a huge amount of issues related to the fact that they have a huge new rewrite deprecating old versions while the new version is not ready: entirely new ways of doing things with missing documentation and example…
Your experience mirrors my experience quite a bit. I remember a few years ago playing with ANTLR precisely as an alternative to flex/bison for a project I was working on. Just recently I had an embedded project (quite resource constrained, like 4kB of RAM) that needed to parse some data coming in a serial port, and I thought “hey, I should look at ANTLR again”.
I fire it up, play around a bit, and did a calculator example or something like that. The next step, I figure, is to get it to generate some C code so I can test it on the device. Turns out ANTLR4 has dropped the C backend entirely! C++ is, currently, a no-go for the project, so... I guess I’m out of luck there.
Pleasantly, in a discussion with a friend, he asked the silly question: “couldn’t you parse those strings with sscanf()?”. I blinked in disbelief, wrote the tiniest parser to split the input on newlines, and sscanf() did the trick.
Re: Why not to use (f)lex, yacc or bison
#85Earlier quoted context omitted.
I first learned Antlr way back about 20 years ago and found it fairly impressive. So when I go to write a new parser or lexer I always go take a look again... But every time I take a new look at Antlr there's a huge amount of issues related to the fact that they have a huge new rewrite deprecating old versions while the new version is not ready: entirely new ways of doing things with missing documentation and example…
Your experience mirrors my experience quite a bit. I remember a few years ago playing with ANTLR precisely as an alternative to flex/bison for a project I was working on. Just recently I had an embedded project (quite resource constrained, like 4kB of RAM) that needed to parse some data coming in a serial port, and I thought “hey, I should look at ANTLR again”. I fire it up, play around a bit, and did a calculator ex…
Yeah sscanf can do quite a bit. But it doesn't have any informative error messages, can't recover (skip to the next semicolon or whatever), doesn't keep a symbol table, can't do any parsing more complex than "the next line looks exactly like this pattern" etc.
So after years of rolling my own, I finally decided I would never do that again. All the off-by-one errors, difficult-to-diagnose failures and endless fiddling is out of my life now. I just use a parser tool.
Re: Why not to use (f)lex, yacc or bison
#86Earlier quoted context omitted.
In my professional experience I build parsers, like at least 10 per year and 99% are written using ANTLR. I did not find a language that I was not able to parse with it, so far. I think it does make sense to write manually parsers for performance and error messages, but it should be clear that this means raising cost by ~10 times. It is worth the effort if you are building a compiler for Java, for example, probably n…
> I did not find a language that I was not able to parse with [ANTLR], so far. I have plenty. Publish contact info on your HN profile and I can send some.
you can contact me at federico -> tomassetti.me
Re: Why not to use (f)lex, yacc or bison
#87Earlier quoted context omitted.
I am sorry we had this impression. Sure, we use ANTLR a lot, for commercial and open-source projects, and we had to work with parsers written in (flex), yacc, and bison, so we shared our experiences as people who work all day long with parsers. We offer a lot of free resources on ANTLR and we have not specific interest in advertising ANTLR. For us it is a tool we use and love, just that
Almost anything fares better than flex/bison, this is shooting the proverbial fish in a barrel. It comes across very strangely that your brother writes as if y'all are unaware of the existence of parser generators other than ANTLR. This makes me sad. In a fair comparison, ANTLR really is not a great tool; its capabilities are eclipsed by its marketing.
We have several articles comparing tens of other tools. That said of all the tools we worked with, ANTLR was the one we were most efficient with. It is true, there is a learning curve, but in our experience learning it was a great investment. I can understand other could have different preferences
Re: Why not to use (f)lex, yacc or bison
#88Re: Why not to use (f)lex, yacc or bison
#89I have seen multiple projects that run into maintenance problems due to the large grammar with thousands lines and antlr 3/4 incompatibility. Generally parser generator adds a layer of complexity/constraint which may be significant if you need full control of lexing, parsing, semantic processing, error recovery of your language. After working on a few language projects (including core language, web-based editor with…
I am a bit surprised by this, as I had the opposite experience. I find ANTLR grammars much more maintanable than the hand-written parsers I encountered. Indeed I was asked to port hand-written parsers to ANTLR for maintanability. Also, ANTLR4 seems to me to produce grammars which are clearer than ANTLR3. The weak point is error recovering, in my opinion. While ANTLR offers a sort-of-decent error recovery strategy for…
My experience has been similar to yours. The company I work at has very complex business logic written as a custom DSL parsed/executed by a crazy pile of Perl code.
Recently we had requirements to come up with a way of providing on-the-fly validation in a web editor. This was only possible by ditching the old implementation and re-writing the grammar using ANTLR. While the old implementation is unmaintainable and (probably, who knows?) buggy, the ANTLR implementation is trivial to work on, test and add new features to.
If you're working with a limited time budget, which is common when your main job isn't to maintain the language, then parser generators such as ANTLR are a godsend. ANTLR even enables you to generate parsers in different languages depending on where it needs to be executed. Need something to run client-side, in the users browser? Generate a JavaScript parser and you're done. Need it to run in the Java backend? Generate it in Java and call it a day.
While it's true that error handling isn't the best, it's already better than nothing, and, as you say, can probably be improved with a bit of customization.
Re: Why not to use (f)lex, yacc or bison
#90Earlier quoted context omitted.
Both go and OCaml use LR parser generators, precisely because they make it easy to give excellent error messages. Basically the key insight is that you can use the parsing automaton state to classify syntax errors, which makes it as easy as falling off a log to produce good messages. The paper originating this technique is Clinton Jeffery's TOPLAS 2003 paper “Generating LR Syntax Error Messages from Examples.” This i…
Go doesn't use a generated parser. Both the parser and the lexer are hand-written: https://github.com/golang/go/blob/master/src/go/parser/parse... https://github.com/golang/go/blob/master/src/go/scanner/scan... Why did you think it was generated?
FTR: The go/parser package was _never_ used by the Go compiler. It used to be a yacc generated one when the compiler itself was in C. Then it was all mechanically translated to Go. And finally, a hand written lexer and parser were adopted, but different from the ones in the stdlib.