Earlier quoted context omitted.
Some compilers tokenize while parsing, but for a different reason: it's faster and uses less memory to generate the AST (and sometimes even do analyses) while you're reading the input than to allocate and store a giant list of tokens and then parse that. Most compilers try not to fail when they encounter a syntax error, they try to "recover" and parse the remaining document, usually starting from the next valid state…
> Some compilers tokenize while parsing, but for a different reason: it's faster and uses less memory Rather legit reasons.. The one I'm writing does this. It seems akin to natural language processing. You'd interrupt a speaker early if you can't make sense of his uttering. > Most compilers try not to fail when they encounter a syntax error, they try to "recover" and parse the remaining document This seems ardous for…
But if you're reading a book and can't understand a sentence, you'll probably glance at the rest of the page to see if there are clues in the context.
Maybe it's a printing error and a paragraph has been repeated. A human who sees the entire page will detect that problem immediately and simply skip over the repeated paragraph. A computer that gives up at the first sight of incongruency has no idea that recovery was so easy.
The analogy here might be relevant to error messages. A compiler that "sees the whole page" can potentially offer more useful suggestions to the programmer about how to fix a problem.