Why not to use (f)lex, yacc or bison • Hacker News Board

Re: Why not to use (f)lex, yacc or bison

#71

Post by minieggs » Thu, Mar 05, 2020, 7:37 PM UTC

atombender wrote: post #69

minieggs wrote: post #68

Earlier quoted context omitted.

If you look closely at the Go code you'll see it was once C code generated by some parser generater (which I assume was Bison). Believe it was Russ Cox that wrote a program that transformed the generated C code into Go. Edit: I forked the compiler a while ago to add maybe types (a la Rust result). Looks like the compiler code has changed quite a bit from when I was playing w/ it.

That's a long time ago. You can see vestiges of it here and there (mostly in deeper areas of the compiler, which goes all the way back to Plan 9), but the parser now looks very clean.

Very true and agree. Parser is in good shape.

Re: Why not to use (f)lex, yacc or bison

#72

Post by jasonlhy » Thu, Mar 05, 2020, 9:35 PM UTC

I think they are learning tools for people to start and to play around. It is hard for an application developer with no background knowledge to understand what a parser does because it is quite abstract, for example lexer, tokens and expression tree.

However, to build a realistic and comprehensive compiler, I believe to build it manually is a much better option because it is less error prone, more flexible to tweak around and favours unit testing. Perhaps parser generator is better? I am investigating these kind of tools for work because we need to make our own unique formula implementation. I did use yacc for a school project, and I know its limitation. Since I am busy at my staff so I just let my colleague to make their decision and they decide to use jison. It turns out the product owners want see a much better error message in invalidating the formula, also we need to define our own function so we have to switch to another implementation in the next phrase.

Re: Why not to use (f)lex, yacc or bison

#73

Post by ftomassetti » Fri, Mar 06, 2020, 8:19 AM UTC

commandlinefan wrote: post #54

kqr wrote: post #17

Is it not the case that most serious parser implementations are hand-written? In part because it makes it so much easier to provide good diagnostic messages. I feel like every other project moves from parser-generators to hand-rolled parsers and levels.

That’s what I expected this article to be about - it looks like it’s actually a long-form sales pitch for ANTLR training (but still worth a read). I’ve never been fortunate (or unfortunate) enough to be called on to put together a real compiler since school, but I’ve spent a lot of time working with “lite” parsers like JAXB and Hibernate (or variants thereof), and I’ve come to the same conclusion: I’m better off just…

In my professional experience I build parsers, like at least 10 per year and 99% are written using ANTLR. I did not find a language that I was not able to parse with it, so far.

I think it does make sense to write manually parsers for performance and error messages, but it should be clear that this means raising cost by ~10 times. It is worth the effort if you are building a compiler for Java, for example, probably not if you want to process some DSL you developed.

Disclaimer: I am the brother of the author of this article

Re: Why not to use (f)lex, yacc or bison

#74

Post by ftomassetti » Fri, Mar 06, 2020, 8:24 AM UTC

brutuz wrote: post #33

I have seen multiple projects that run into maintenance problems due to the large grammar with thousands lines and antlr 3/4 incompatibility. Generally parser generator adds a layer of complexity/constraint which may be significant if you need full control of lexing, parsing, semantic processing, error recovery of your language. After working on a few language projects (including core language, web-based editor with…

I am a bit surprised by this, as I had the opposite experience. I find ANTLR grammars much more maintanable than the hand-written parsers I encountered. Indeed I was asked to port hand-written parsers to ANTLR for maintanability. Also, ANTLR4 seems to me to produce grammars which are clearer than ANTLR3.

The weak point is error recovering, in my opinion. While ANTLR offers a sort-of-decent error recovery strategy for free, one has to customize it if they want to get great error messages.

I think that many maintainability issues are due to poor usage of ANTLR. What we do is to: 1) Limit semantic actions 2) For complex languages do tree-transformations, after parsing

With this approach we got pretty decent results.

Personally I find hand-rolled parsers too costly to build and harder to maintain, but I have limited experience with them. I guess it also depends on the context: if you are designing a DSL while writing the parser you want to be able to evolve it very quickly and in that context I think that a parser generator is very useful. If instead I would need to build an industrial grade parser for a general purpose language, having a large budget, then I would go for an hand-rolled parser

Re: Why not to use (f)lex, yacc or bison

#75

Post by ftomassetti » Fri, Mar 06, 2020, 8:26 AM UTC

nottorp wrote: post #37

"ANTLR instead is more actively developed. It has been re-written from scratch a few times during the years, so the code has is of good quality." ... rewrites improve code quality ... right ... right???

Rewriting perhaps not, but being able to fix the ebnf-flavor used to make it more powerful it is a great thing. If you have to stay 100% compatible with all the mistakes accumulated in 30 years you are tied

Re: Why not to use (f)lex, yacc or bison

#76

Post by ftomassetti » Fri, Mar 06, 2020, 8:30 AM UTC

hoseja wrote: post #43

This is just an advert for ANTLR. I have no real experience with either but the article is really grating to read beyond the introduction.

I am sorry we had this impression. Sure, we use ANTLR a lot, for commercial and open-source projects, and we had to work with parsers written in (flex), yacc, and bison, so we shared our experiences as people who work all day long with parsers. We offer a lot of free resources on ANTLR and we have not specific interest in advertising ANTLR. For us it is a tool we use and love, just that

Re: Why not to use (f)lex, yacc or bison

#77

Post by ftomassetti » Fri, Mar 06, 2020, 8:32 AM UTC

kqr wrote: post #17

Is it not the case that most serious parser implementations are hand-written? In part because it makes it so much easier to provide good diagnostic messages. I feel like every other project moves from parser-generators to hand-rolled parsers and levels.

I think that hand-rolled parsers are good if: 1) You have a team that knows well what they are doing 2) You have a lot of resources 3) Your language is stable

I would argue that parser-generators are what made those projects to be started and prosper in the first place. Then once one is successful a custom solution could make sense but in my experience it is more expensive and potentially less maintanable, unless you know very welll your way around parsers and language tooling

Re: Why not to use (f)lex, yacc or bison

#78

Post by ftomassetti » Fri, Mar 06, 2020, 8:36 AM UTC

xalqpp wrote: post #14

This whole article is very polemic and promotes an ANTLR book. It is futile to comment in detail because nearly every statement is false.

..wow, this is a bit strong. I would encourage you to be more respectful of persons who work in this very specific field and share their ideas. There are always persons behind some work you insult so easily.

You know, there are over 2M persons who read our articles. A few hundred also bought a book or a video-course from us, but the vast majority just got some information from free, and we like it in this way. I do not think we got so many persons interested in what we do by lying.

If you have a different professional experience I would happy to learn from it.

Re: Why not to use (f)lex, yacc or bison

#79

Post by bmn__ » Fri, Mar 06, 2020, 9:38 AM UTC

ftomassetti wrote: post #73

commandlinefan wrote: post #54

Earlier quoted context omitted.

That’s what I expected this article to be about - it looks like it’s actually a long-form sales pitch for ANTLR training (but still worth a read). I’ve never been fortunate (or unfortunate) enough to be called on to put together a real compiler since school, but I’ve spent a lot of time working with “lite” parsers like JAXB and Hibernate (or variants thereof), and I’ve come to the same conclusion: I’m better off just…

In my professional experience I build parsers, like at least 10 per year and 99% are written using ANTLR. I did not find a language that I was not able to parse with it, so far. I think it does make sense to write manually parsers for performance and error messages, but it should be clear that this means raising cost by ~10 times. It is worth the effort if you are building a compiler for Java, for example, probably n…

> I did not find a language that I was not able to parse with [ANTLR], so far.

I have plenty. Publish contact info on your HN profile and I can send some.

Re: Why not to use (f)lex, yacc or bison

#80

Post by bmn__ » Fri, Mar 06, 2020, 9:38 AM UTC

ftomassetti wrote: post #76

hoseja wrote: post #43

This is just an advert for ANTLR. I have no real experience with either but the article is really grating to read beyond the introduction.

I am sorry we had this impression. Sure, we use ANTLR a lot, for commercial and open-source projects, and we had to work with parsers written in (flex), yacc, and bison, so we shared our experiences as people who work all day long with parsers. We offer a lot of free resources on ANTLR and we have not specific interest in advertising ANTLR. For us it is a tool we use and love, just that

Almost anything fares better than flex/bison, this is shooting the proverbial fish in a barrel.

It comes across very strangely that your brother writes as if y'all are unaware of the existence of parser generators other than ANTLR. This makes me sad. In a fair comparison, ANTLR really is not a great tool; its capabilities are eclipsed by its marketing.