The Super Tiny Compiler

Post by bmn__ » Sat, Mar 04, 2023, 4:04 PM UTC

deterministic wrote: post #40

bmn__ wrote: post #29

Earlier quoted context omitted.

For the year 1970 and after: never. This should always be part of the parser for reasons of being confidently correct. Thinking otherwise just leads to suffering. The problem with tutorials like Tiny Compiler or Crafting Interpreters is that the authors do not run into problems with the code shown in their teaching materials, but as soon as a student wants to apply it to modestly complex grammars, it stops working. T…

I am a professional compiler writer and nope you are wrong. I never tokenise everything up front and instead read tokens on demand. It is much faster and enables me to change how tokens are interpreted on the fly. Which enables me to parse different languages with different token definitions in the same file. Something that I need for the production work I do.

Post by deterministic » Sun, Mar 05, 2023, 2:52 AM UTC

bmn__ wrote: post #41

deterministic wrote: post #40

Earlier quoted context omitted.

I am a professional compiler writer and nope you are wrong. I never tokenise everything up front and instead read tokens on demand. It is much faster and enables me to change how tokens are interpreted on the fly. Which enables me to parse different languages with different token definitions in the same file. Something that I need for the production work I do.

>>> when is it useful, if ever, to tokenize the whole input beforehand >> never > nope you are wrong. I never tokenise everything up front Explain how you could possibly arrive at the complete opposite meaning of what I wrote when you even use the same word "never" as I did? Quick, get another downvote in, that'll teach me! Peak HN moment here.