Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's write a treesitter major mode for Emacs

masteringemacs.org

Post reply on HN
Print view View original

71–80 of 86 posts

Re: Let's write a treesitter major mode for Emacs

#71

Post by Brentward »

shanusmagnus wrote: post #53

Is there anything that returns a parse tree of an org document? A while ago I wrote some super hacky elisp to navigate around the structure of a giant org mode doc, but it was rickety and terrible and constantly breaking. Part of this is surely that I don't know wtf I'm doing, but it seemed like there was not an underlying data structure held in memory that you could conveniently query / manipulate, but rather, most…

In org-alert we use `org-map-entries` and a simple `org-alert--parse-entry` function for stripping out the details we're looking for. Depending on what you want, it's not exactly a data structure, but maybe it will help you get started!

https://github.com/spegoraro/org-alert/blob/master/org-alert...

Re: Let's write a treesitter major mode for Emacs

#72

Post by mdaniel »

Difwif wrote: post #44
BaculumMeumEst wrote: post #38

Earlier quoted context omitted.

they are mostly used for different things. lsp (and lsp-mode) are mostly concerned with IDE functionality- go-to definition, show references, displaying project errors in real time without explicitly building, etc. tree-sitter builds a syntax tree of your source code; its applications are things like syntax highlighting and structural navigation of your code. there is some overlap in functionality, lsp has somewhat s…

Quick follow up, looked into the LSP spec and it does define "SemanticTokens" since 3.16 https://microsoft.github.io/language-server-protocol/specifi... Obviously it depends on the LSP implementation but it looks expressive enough to replace treesitter?

That was interesting, thanks for pointing it out

I was tremendously sad to see that the Typescript Language Server wasn't owned by Microsoft https://microsoft.github.io/language-server-protocol/impleme...>, since if there was any sanity in the world a spec bump would travel with a reference implementation showing how they envision such a thing being used

But, I found that the Typescript Language Server that they did list does indeed have a semantic-tokens module in it, although it's much shorter than I would have expected from reading that section in the spec: https://github.com/typescript-language-server/typescript-lan...

Re: Let's write a treesitter major mode for Emacs

#73

Post by tuukkah »

nerdponx wrote: post #66
tuukkah wrote: post #24

Earlier quoted context omitted.

Insightful article. Key point: > Fourth, Microsoft itself doesn’t try to take advantage of M + N. There’s no universal LSP implementation in VS Code. Instead, each language is required to have a dedicated plugin with physically independent implementations of LSP. There's no (official) LSP implementation for Typescript either. Instead of using LSP, Microsoft maintains tsserver which uses a custom protocol for better i…

But Tree Sitter is an M+N tool. Also LSP was never going to completely eliminate the need for language-specific plugins. But it does significantly reduce the amount of work needed to build each one, providing a consistent baseline.

In my classification, Tree-sitter is a building block (a parser), not a tool (compiler, editor, highlighter etc.) And as seen in this HN submission about implementing HTML support for Emacs, it's used in M×N integrations.

I didn't know tree-sitter-highlight exists though. It seems to provide M+N highlighting and includes a CLI tool. https://tree-sitter.github.io/tree-sitter/syntax-highlightin...

Similarly, tree-sitter tags seems to provide M+N code indexing: https://tree-sitter.github.io/tree-sitter/code-navigation-sy...

Re: Let's write a treesitter major mode for Emacs

#74

Post by tuukkah »

mdaniel wrote: post #72
Difwif wrote: post #44

Earlier quoted context omitted.

Quick follow up, looked into the LSP spec and it does define "SemanticTokens" since 3.16 https://microsoft.github.io/language-server-protocol/specifi... Obviously it depends on the LSP implementation but it looks expressive enough to replace treesitter?

That was interesting, thanks for pointing it out I was tremendously sad to see that the Typescript Language Server wasn't owned by Microsoft https://microsoft.github.io/language-server-protocol/impleme... >, since if there was any sanity in the world a spec bump would travel with a reference implementation showing how they envision such a thing being used But, I found that the Typescript Language Server that they did…

That code implements a (luckily) simple transformation from Microsoft's tsserver format to Microsoft's LSP format:

> Transforms the semantic token spans given by the ts-server into lsp compatible spans

Re: Let's write a treesitter major mode for Emacs

#75

Post by confounded »

mark_l_watson wrote: post #68

This is from the author of the excellent book Mastering Emacs. I am very far from being knowledgeable about programming on the Emacs platform, but I am trying to learn. I grabbed the name M-x-AI.com a while back with the goal of integrating other people’s Emacs packages with some of my own hacks into a better AI dev work environment and writing a short book on it. I have been using Emacs since, I think, 1982. There a…

Out of interest, do you use Emacs as an alternative to Jupyter for interactive work (examining plots etc.)?

If so, which modes and packages do you use?

Re: Let's write a treesitter major mode for Emacs

#76

Post by tuukkah »

mbork_pl wrote: post #45
raincole wrote: post #2

I'm not trying to bash Emacs or treesitter or anyone. But I find it mildly amusing that after so many decades, parsing and syntax highlighting aren't a perfectly solved problem, considering programming languages are the most used tools for developers.

Well, there are more problems like that. You'd think diffing is a solved problem, and yet we still struggle with syntax-aware diffs (I use difftastic, which is great, but doesn't always work well, and is under constant development).

Interesting, it's also based on parsing by Tree-sitter: https://difftastic.wilfred.me.uk/parsing.html

Re: Let's write a treesitter major mode for Emacs

#77

Post by nerdponx »

tuukkah wrote: post #73
nerdponx wrote: post #66

Earlier quoted context omitted.

But Tree Sitter is an M+N tool. Also LSP was never going to completely eliminate the need for language-specific plugins. But it does significantly reduce the amount of work needed to build each one, providing a consistent baseline.

In my classification, Tree-sitter is a building block (a parser), not a tool (compiler, editor, highlighter etc.) And as seen in this HN submission about implementing HTML support for Emacs , it's used in M×N integrations. I didn't know tree-sitter-highlight exists though. It seems to provide M+N highlighting and includes a CLI tool. https://tree-sitter.github.io/tree-sitter/syntax-highlightin... Similarly, tree-sitt…

It's an M+N tool in my opinion because it parses M languages with a uniform interface, for use in N higher-level tools. I can use the same Python grammar/parser with any tool that's based on Tree Sitter.

Re: Let's write a treesitter major mode for Emacs

#78

Post by jlokier »

vidarh wrote: post #23
raincole wrote: post #2

I'm not trying to bash Emacs or treesitter or anyone. But I find it mildly amusing that after so many decades, parsing and syntax highlighting aren't a perfectly solved problem, considering programming languages are the most used tools for developers.

Parsing of a correct program is a pretty "solved" problem. But fast enough re-parsing of fragments and recovery from errors is a much more complex problem, that often doesn't have a single correct answer, and it's also a much newer problem in as much as syntax-highlighting is much newer feature, being preceded largely by "offline" pretty-printers with very different constraints. The extent to which modern compilers t…

> it's also a much newer problem in as much as syntax-highlighting is much newer feature

Just an aside. I wouldn't associate syntax highlighting with new. It's almost as old as text editors. Accurate syntax highlighting in real-time visual text editors is over 40 years old by now and was becoming common by the late 1980s.

GNU Emacs gained syntax highlighting in 1989 and it was considered late to the party. Many programming editors and IDEs had syntax highlighting by then.

Re: Let's write a treesitter major mode for Emacs

#79

Post by bch »

vkazanov wrote: post #25
aidos wrote: post #22

Earlier quoted context omitted.

Interestingly, treesitter is designed for this space (parsing invalid structures with a time component) but it’s still not used as a base for LSPs. I once asked why on HN and people that know more than me said it wasn’t suitable.

tree-sitter definitely can be used as a incremental parser for an LSP. But the original purpose was and still is to provide a standard format for an editor-agnostic way to parse languages. As somebody who wrote a tree-sitter grammar recently I can confirm that the library definitely has its share of... Interesting choices. But there's nothing else like it as most parser generators are non-incremental, don't do genera…

> has its share of… Interesting choices

Curious to hear more.

Re: Let's write a treesitter major mode for Emacs

#80

Post by troupo »

craftkiller wrote: post #69
troupo wrote: post #19

Earlier quoted context omitted.

Syntax errors are not limited to just typing a quote

I never said they were. I'm just pointing out that we already experience the same problem in the case of quotes without the world ending, and I don't think expanding the problem to all syntax errors will make it much worse.

It would make it significantly much worse. Because every incomplete statement that you're just writing would break all syntax highlighting right after your cursor. And if you use any modern editor or IDE this doesn't happen.
Post reply on HN