The naked truth about writing a programming language (2014)

This advice is quite good for certain types of programming language, if you look at the world the same was as he does. We're implementing Dark from a completely different worldview, and from that vantage point, a lot of these aren't exactly wrong, but different.

> Syntax matters

The approach we took with Dark is that there isn't a syntax, per se, in that there isn't a parser. There's certainly a textual view of Dark and so it's important that the code looks good (it currently looks only OK, in my opinion). But as a result, we have other options for minimizing keystrokes (autocomplete built-in), parsing (again, no parse), and minimizing keywords (you're allowed have a variant with the same name as a keyword, which isn't allowed in languages with parsers (well, lexers, but same idea).

> context free grammars

No parser, no need to have a grammar. His point about IDEs is great - we only support our own IDE (a controversial decision to be sure!)

> Redundancy

This is an esoteric parsing problem, that only applies if you have a parser. No parser means no syntax errors. We are left with editor errors (how does the editor have good UX) and run-time errors.

> Implementation

He's right about how hard error messages are, so I feel good about our "not parsing" approach.

> Compiler speed

Our compilation is instant. The way it's instant is:

- very small compilation units: you're editing a single function at a time, and so nothing else needs to be parsed.

- no parser: The editor directly updates the AST, so you don't have to read the whole file (there isn't a "file" concept). Even in JS, that means an update takes a few milliseconds at the most.

- extremely incremental compilation: making a change in the editor only changes the exact AST construct that's changing.

> Lowering

This is really about compilation. One thing you can do, which is what we do, is have an interpreter. Now, interpreters are slow, but we simply have a different goal with the language, which is to run HTTP requests quickly. We do run into problems with the limit of the interpreter, but we plan to add a compiler later to deal with this.

Really what I'm suggesting here is that compiled languages look at having interpreters in addition to compilers.

> i/o performance

> memory allocation

I think he's approaching this with an implicit goal of "it must be as fast as possible", which isn't necessarily a goal for all languages.

> You’ve done it, you’ve got a great prototype of the new language. Now what? Next comes the hardest part. This is where most new languages fail. You’ll be doing what every nascent rock band does — play shopping malls, high school dances, dive bars, etc., slowly building up an audience. For languages, this means preparing presentations, articles, tutorials, and books on the language. Then, going to programmer meetings, conferences, companies, anywhere they’ll have you, and show it off. You’ll get used to public speaking, and even find you enjoy it (I enjoy it a lot).

Well this is certainly correct!

I don't think that comment was about reader macros. I don't see reader macros used much, while normal macros are frequently used to create new control structures.

This point in the article is pretty weird (the rest is good). I don't think IDEs do a lot of parsing to do syntax highlighting, isn't it all just regex matching to identify the types of tokens? I'd be interested in real-world examples of IDEs doing something more complex to achieve syntax highlighting. And conversely, examples of imperfect syntax highlighting of C++ due to the undecidability of its input language. I mean, yes, an actual parser for C++ must be able to execute template computations, and that's a pain, but why would that be relevant to syntax highlighting? It isn't.

Now, there are valid reasons for running a proper language frontend from the IDE: Error reporting of all kinds, not just syntax but also type errors and whatever else the compiler likes to complain about. But there the IDE should not try to replicate the parser. Parsing only a context-free surface language will not catch type errors, exactly because enforcing a static type system requires one to "look things up in a symbol table". So the actual compiler should provide an IDE-friendly mode where it runs its frontend and reports errors back, and the IDE should not try to roll its own version of this.

In other words, the IDE is not relevant either way.

> Any programming language that is massively adopted is context free?

Statically typed languages require semantic analysis and symbol table lookups, so they are out.

And yet we have context-free grammars for all widely adopted statically typed programming languages. And we also have additional semantic checks for all of them. The notion of a "context free programming language" being one that has context free syntax and no semantic checks at all is not a useful notion. Don't worry about it while designing your language.

> I don't think IDEs do a lot of parsing to do syntax highlighting, isn't it all just regex matching to identify the types of tokens?

IntelliJ highlights variables etc. differently if they're unused and can highlight different identifiers in different colors. Not sure about other current IDEs.

That said I never felt like I needed such features, normal syntax highlighting does the job pretty well for me.