How to implement a programming language in JavaScript

It would be nice when saying things like "don't use regexps for parsing" that it is accompanied by an explanation (possibly I didn't read far enough to see the explanation...)

There are only a few actual computer science theory things that I think all programmers should know and this is one of them. There are classifications of grammars (1). Without going into detail, regular expressions can only be used to parse regular grammars.

The problem is that most grammars for programming languages, file formats, communication protocols, etc, are not regular grammars. With a regular grammar you can look at the current state and the next input symbol to determine the next state. With context free grammars you need to have a stack of states. With context sensitive grammars you need to have a stack of stacks states. With unrestricted grammars you are essentially screwed ;-)

So your first task when you are designing a language or file format or communication protocol or whatever is to choose a simple grammar. If you choose a regular grammar then you can (and probably should) use regular expressions to parse it. With context free or context sensitive grammars you can often do lexical analysis (creating symbols that you then pass to your parser) with regular expressions, but you need something more complex for parsing the stream of symbols (i.e., you need to be able to put parser states on a stack).

The problem that you often see is that people design things and have absolutely no idea what kind of grammar it is. They use regular expressions (or some ad-hoc code) and then try to keep track of state in global variables. If they happen to have a context sensitive grammar then chances are their parser simply will not work correctly no matter what they do.

You may be wondering why people choose to use more complex grammars if it is harder to parse. The main reason is that complex grammars give you more options for expression. Sometimes it is extremely difficult or even impossible to represent something with a regular grammar. Having said that, though, you should almost always try to keep your grammars at least context free. Once you get into context sensitive grammars, the difficulty of parsing will either make your parser very difficult to implement or buggy as hell (usually both). Usually it is better to remove functionality than it is to move from a context sensitive to context free grammar.

In the past I have often seen file formats that have unrestricted grammars (converting file formats used to be my job). People who do this should be replaced with programmers who know what they are doing :-P

(1) - https://en.wikipedia.org/wiki/Chomsky_hierarchy

It used to be written in C before bootstrapping.

This is also a really good resource: https://github.com/thejameskyle/the-super-tiny-compiler

This is all true, for sure, but I'd like to point out that PEGs are a formalism that isn't much harder to learn than regular expressions, and totally appropriate for parsing a fairly large class of languages. I've also seen PEG-based (pakrat) parser generators for just about every programming language (some better than others, of course). Again, these tend to be not much harder to learn than regular expressions, and because of the "ordered choice" operator, many programmers seem more comfortable reasoning about PEGs than a Yacc specificiation, for example.

PEGs aren't the only formalism that attempts to make writing certain classes of parsers easy. You can get a long way with parser combinators, and Matthew Might's "Yacc is Dead" paper discusses another (extremely general) approach, which I have unfortunately not seen many implementations of.

Anyway, enough of an addendum! Keep on spreading the Good Word ;)

Not only FP. Lots of languages are self-hosted. So actually most modern parsers are not written in C, since they are written in the language they parse.