Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's write a treesitter major mode for Emacs

masteringemacs.org

Post reply on HN
Print view View original

21–30 of 86 posts

Re: Let's write a treesitter major mode for Emacs

#21

Post by lelanthran »

pawelmurias wrote: post #11
craftkiller wrote: post #8

Earlier quoted context omitted.

I'm not the GP, but personally I'd consider syntax highlighting perfectly solved if it was handled in the language server. That way we could have highlighting performed by 100% accurate parsers instead of approximations. It wouldn't be as fast as in-editor simple tree-sitter highlighting, but I don't see any issue with my text appearing uncolored at first and gaining color as my editor polls the language server as I…

You would only get the top half of your code highlighted because a compiler doesnt usually continue when encountering syntax errors due to partial code

> You would only get the top half of your code highlighted because a compiler doesnt usually continue when encountering syntax errors due to partial code

Sounds like a feature - don't need to scan for tiny squiggly red lines, or examine some tiny font output in a status bar somewhere.

I'd love it.

Re: Let's write a treesitter major mode for Emacs

#22

Post by aidos »

troupo wrote: post #4
raincole wrote: post #2

I'm not trying to bash Emacs or treesitter or anyone. But I find it mildly amusing that after so many decades, parsing and syntax highlighting aren't a perfectly solved problem, considering programming languages are the most used tools for developers.

The problem is that a good editor-compatible tool for parsing and syntax highlighting is at cross-purposes with what you want from a compiler. A good overview here: https://matklad.github.io/2022/04/25/why-lsp.html

Interestingly, treesitter is designed for this space (parsing invalid structures with a time component) but it’s still not used as a base for LSPs. I once asked why on HN and people that know more than me said it wasn’t suitable.

Re: Let's write a treesitter major mode for Emacs

#23

Post by vidarh »

raincole wrote: post #2

I'm not trying to bash Emacs or treesitter or anyone. But I find it mildly amusing that after so many decades, parsing and syntax highlighting aren't a perfectly solved problem, considering programming languages are the most used tools for developers.

Parsing of a correct program is a pretty "solved" problem.

But fast enough re-parsing of fragments and recovery from errors is a much more complex problem, that often doesn't have a single correct answer, and it's also a much newer problem in as much as syntax-highlighting is much newer feature, being preceded largely by "offline" pretty-printers with very different constraints.

The extent to which modern compilers try to parse past errors still varies greatly, with a whole lot not even trying to.

But just any recovering parser also does not mean the problem is solved. E.g. you've typed "foo". Now you type "(". It'd be very annoying if your editor now re-colors everything as an error, so you typically want some error recovery. But how soon? Do you assume the tokens immediate afterwards are par of what was a valid expression until you typed "foo", or are they a valid part of an argument list? And where do they end? Do you just delay re-parsing until the user has typed more? Or left the line? Sometimes that can help, sometimes it will just make things worse.

Parsing methods that work fine if you assume you can "reset" the parse at many different points which tend to constrain the area considered an error and so reducing the size of a typical re-parse will fail badly if you want stricter re-parsing that frequently may trigger reparsing most of a file, for example.

A lot of this is subjective, and picking the "right" way of handling it largely comes down to unpacking humans unstated preferences, and trying to reconcile competing and possibly contradictory preferences.

Re: Let's write a treesitter major mode for Emacs

#24

Post by tuukkah »

troupo wrote: post #4
raincole wrote: post #2

I'm not trying to bash Emacs or treesitter or anyone. But I find it mildly amusing that after so many decades, parsing and syntax highlighting aren't a perfectly solved problem, considering programming languages are the most used tools for developers.

The problem is that a good editor-compatible tool for parsing and syntax highlighting is at cross-purposes with what you want from a compiler. A good overview here: https://matklad.github.io/2022/04/25/why-lsp.html

Insightful article. Key point:

> Fourth, Microsoft itself doesn’t try to take advantage of M + N. There’s no universal LSP implementation in VS Code. Instead, each language is required to have a dedicated plugin with physically independent implementations of LSP.

There's no (official) LSP implementation for Typescript either. Instead of using LSP, Microsoft maintains tsserver which uses a custom protocol for better integration.

Take-home message: don't try to write a universal tool that solves everything, as it will be lowest common denominator. Create building blocks such as Tree-sitter, which make specific (M×N) integrations easier and more powerful.

Re: Let's write a treesitter major mode for Emacs

#25

Post by vkazanov »

aidos wrote: post #22
troupo wrote: post #4

Earlier quoted context omitted.

The problem is that a good editor-compatible tool for parsing and syntax highlighting is at cross-purposes with what you want from a compiler. A good overview here: https://matklad.github.io/2022/04/25/why-lsp.html

Interestingly, treesitter is designed for this space (parsing invalid structures with a time component) but it’s still not used as a base for LSPs. I once asked why on HN and people that know more than me said it wasn’t suitable.

tree-sitter definitely can be used as a incremental parser for an LSP. But the original purpose was and still is to provide a standard format for an editor-agnostic way to parse languages.

As somebody who wrote a tree-sitter grammar recently I can confirm that the library definitely has its share of... Interesting choices. But there's nothing else like it as most parser generators are non-incremental, don't do generalized parsing and don't provide decent error recovery.

Re: Let's write a treesitter major mode for Emacs

#26

Post by nanna »

G3rn0ti wrote: post #17

BTW: While Emacs 29.1 comes with "treesitter" built-in, you still need to manually build and install any treesitter language plugin implementing the actual language specific parser. This can be fiddly and frustrating doing it yourself. I had a quick success with using this convenience script: https://github.com/casouri/tree-sitter-module/ . It provides fully-automated builds for the most popular languages (including…

Technically in Emacs 29.1 tree-sitter is still only an optional build option, which a given package maintainer may have 'built in' to your package. It isn't actually a default. If you build it from source you need to pass the --with-tree-sitter flag to ./configure. See:

https://www.masteringemacs.org/article/how-to-get-started-tr...

What I read from this is that tree-sitter isn't considered quite ready by the Emacs maintainers, perhaps because of the restricted number of actual treesitter modes, or maybe because the treesitter support itself is not quite considered there yet?

Re: Let's write a treesitter major mode for Emacs

#27

Post by globular-toast »

morelisp wrote: post #10
craftkiller wrote: post #8

Earlier quoted context omitted.

I'm not the GP, but personally I'd consider syntax highlighting perfectly solved if it was handled in the language server. That way we could have highlighting performed by 100% accurate parsers instead of approximations. It wouldn't be as fast as in-editor simple tree-sitter highlighting, but I don't see any issue with my text appearing uncolored at first and gaining color as my editor polls the language server as I…

But you want syntax highlighting performed by approximations, because when editing your code is only approximately valid.

It would be even better if the code was always valid because we only edit the AST itself, like what paredit does for Lisp modes.

Re: Let's write a treesitter major mode for Emacs

#28

Post by reuben364 »

craftkiller wrote: post #8
thih9 wrote: post #3

Earlier quoted context omitted.

What would you consider a perfectly solved problem in this case? I.e. how is current development experience bad and how it could be better?

I'm not the GP, but personally I'd consider syntax highlighting perfectly solved if it was handled in the language server. That way we could have highlighting performed by 100% accurate parsers instead of approximations. It wouldn't be as fast as in-editor simple tree-sitter highlighting, but I don't see any issue with my text appearing uncolored at first and gaining color as my editor polls the language server as I…

One interesting instance of semantic highlighting occurs in the Lean 4 language. It has extensible syntax whose parsers can be expressed in the language itself. Since parsing and evaluation are intertwined, highlighting can only really be effectively achieved in an LSP lest you re-implement the entirety of Lean.

Re: Let's write a treesitter major mode for Emacs

#29

Post by morelisp »

globular-toast wrote: post #27
morelisp wrote: post #10

Earlier quoted context omitted.

But you want syntax highlighting performed by approximations, because when editing your code is only approximately valid.

It would be even better if the code was always valid because we only edit the AST itself, like what paredit does for Lisp modes.

No.

(The code being valid s-sexps doesn’t even guarantee it’s syntactically valid after macro expansion, but even if it does this is a bad idea.)

Re: Let's write a treesitter major mode for Emacs

#30

Post by 3836293648 »

nanna wrote: post #26
G3rn0ti wrote: post #17

BTW: While Emacs 29.1 comes with "treesitter" built-in, you still need to manually build and install any treesitter language plugin implementing the actual language specific parser. This can be fiddly and frustrating doing it yourself. I had a quick success with using this convenience script: https://github.com/casouri/tree-sitter-module/ . It provides fully-automated builds for the most popular languages (including…

Technically in Emacs 29.1 tree-sitter is still only an optional build option, which a given package maintainer may have 'built in' to your package. It isn't actually a default. If you build it from source you need to pass the --with-tree-sitter flag to ./configure. See: https://www.masteringemacs.org/article/how-to-get-started-tr... What I read from this is that tree-sitter isn't considered quite ready by the Emacs m…

It's more external dependency. Libjansson (json serialisation) isn't default either and neither is libgccjit (native elisp comp).
Post reply on HN