Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's write a treesitter major mode for Emacs

masteringemacs.org

Post reply on HN
Print view View original

61–70 of 86 posts

Re: Let's write a treesitter major mode for Emacs

#61

Post by ics »

morelisp wrote: post #54
shanusmagnus wrote: post #53

Is there anything that returns a parse tree of an org document? A while ago I wrote some super hacky elisp to navigate around the structure of a giant org mode doc, but it was rickety and terrible and constantly breaking. Part of this is surely that I don't know wtf I'm doing, but it seemed like there was not an underlying data structure held in memory that you could conveniently query / manipulate, but rather, most…

https://orgmode.org/worg/dev/org-element-api.html But even with this I found it pretty awful.

What was awful about it? It's been a while since setting up but my config uses it to pluck the important bits from my Org library into SQLite as I edit; it works well enough and wasn't difficult to set up or understand. Admittedly it is relegated to this one step so that everything else can query docs with SQL but I was quite happy that the API exists so as not to do the parsing myself with the mistakes and minor deviations that would entail.

Re: Let's write a treesitter major mode for Emacs

#62

Post by elviejo »

shanusmagnus wrote: post #53

Is there anything that returns a parse tree of an org document? A while ago I wrote some super hacky elisp to navigate around the structure of a giant org mode doc, but it was rickety and terrible and constantly breaking. Part of this is surely that I don't know wtf I'm doing, but it seemed like there was not an underlying data structure held in memory that you could conveniently query / manipulate, but rather, most…

Organice is org-mode but as react apt. they have pretty complete parser. https://github.com/200ok-ch/organice#background-information

Re: Let's write a treesitter major mode for Emacs

#63

Post by treeblah »

G3rn0ti wrote: post #17

BTW: While Emacs 29.1 comes with "treesitter" built-in, you still need to manually build and install any treesitter language plugin implementing the actual language specific parser. This can be fiddly and frustrating doing it yourself. I had a quick success with using this convenience script: https://github.com/casouri/tree-sitter-module/ . It provides fully-automated builds for the most popular languages (including…

I found this snippet in one of Mickey's earlier tree-sitter posts that works great. It does require searching through the tree-sitter repo to make sure your paths are correct:

  (setq treesit-language-source-alist
      '((typescript "https://github.com/tree-sitter/tree-sitter-typescript" "master" "typescript/src")
        (tsx "https://github.com/tree-sitter/tree-sitter-typescript" "master" "tsx/src")))

  (mapc #'treesit-install-language-grammar (mapcar #'car treesit-language-source-alist))

Re: Let's write a treesitter major mode for Emacs

#64

Post by taeric »

aeonik wrote: post #40
vidarh wrote: post #23

Earlier quoted context omitted.

Parsing of a correct program is a pretty "solved" problem. But fast enough re-parsing of fragments and recovery from errors is a much more complex problem, that often doesn't have a single correct answer, and it's also a much newer problem in as much as syntax-highlighting is much newer feature, being preceded largely by "offline" pretty-printers with very different constraints. The extent to which modern compilers t…

I wonder why there aren't more tools to allow structural editing like Lisp has. This way the file can never be in an unparsable state.

This is not fully correct. You are right that lisp has very regular atoms that you can read, but the full parsing is different. Consider (let (a 0) (1+ a)) is not valid syntax. Such that if you were to add color for the different parts, it would fail. Indeed, if you want to label the parts of the program tree, you have to parse it more than just "lists of atoms."

Re: Let's write a treesitter major mode for Emacs

#65

Post by nerdponx »

raincole wrote: post #2

I'm not trying to bash Emacs or treesitter or anyone. But I find it mildly amusing that after so many decades, parsing and syntax highlighting aren't a perfectly solved problem, considering programming languages are the most used tools for developers.

This is that. Tree Sitter has become one of the foundational advances that is allowing us to make progress on solving that problem.

Re: Let's write a treesitter major mode for Emacs

#66

Post by nerdponx »

tuukkah wrote: post #24
troupo wrote: post #4

Earlier quoted context omitted.

The problem is that a good editor-compatible tool for parsing and syntax highlighting is at cross-purposes with what you want from a compiler. A good overview here: https://matklad.github.io/2022/04/25/why-lsp.html

Insightful article. Key point: > Fourth, Microsoft itself doesn’t try to take advantage of M + N. There’s no universal LSP implementation in VS Code. Instead, each language is required to have a dedicated plugin with physically independent implementations of LSP. There's no (official) LSP implementation for Typescript either. Instead of using LSP, Microsoft maintains tsserver which uses a custom protocol for better i…

But Tree Sitter is an M+N tool.

Also LSP was never going to completely eliminate the need for language-specific plugins. But it does significantly reduce the amount of work needed to build each one, providing a consistent baseline.

Re: Let's write a treesitter major mode for Emacs

#67

Post by hardwaregeek »

raincole wrote: post #2

I'm not trying to bash Emacs or treesitter or anyone. But I find it mildly amusing that after so many decades, parsing and syntax highlighting aren't a perfectly solved problem, considering programming languages are the most used tools for developers.

It's not really a solved problem in general. Most editors appear to use TextMate grammars which nobody likes. Otherwise you have to implement it using whatever custom setup your specific editor uses. It just happens that most languages have some poor soul who set this up already. Emacs is actually on the better side because tree-sitter is a much better setup for writing grammars.

Re: Let's write a treesitter major mode for Emacs

#68

Post by mark_l_watson »

This is from the author of the excellent book Mastering Emacs.

I am very far from being knowledgeable about programming on the Emacs platform, but I am trying to learn. I grabbed the name M-x-AI.com a while back with the goal of integrating other people’s Emacs packages with some of my own hacks into a better AI dev work environment and writing a short book on it. I have been using Emacs since, I think, 1982. There are so many good new packages for integrating CoPilot, GPT-4, etc., as well as major Emacs platform improvements that are too many to list.

Re: Let's write a treesitter major mode for Emacs

#69

Post by craftkiller »

troupo wrote: post #19
craftkiller wrote: post #14

Earlier quoted context omitted.

That's sort-of already the case whenever we start a string in any language that supports multi-line strings. You type the first " and then the rest of your file gets re-interpreted as a string. It is annoying but I still think its a worthy trade-off, because I spend a lot more time reading code than being mid-edit with invalid syntax.

Syntax errors are not limited to just typing a quote

I never said they were. I'm just pointing out that we already experience the same problem in the case of quotes without the world ending, and I don't think expanding the problem to all syntax errors will make it much worse.

Re: Let's write a treesitter major mode for Emacs

#70

Post by Barrin92 »

aeonik wrote: post #40
vidarh wrote: post #23

Earlier quoted context omitted.

Parsing of a correct program is a pretty "solved" problem. But fast enough re-parsing of fragments and recovery from errors is a much more complex problem, that often doesn't have a single correct answer, and it's also a much newer problem in as much as syntax-highlighting is much newer feature, being preceded largely by "offline" pretty-printers with very different constraints. The extent to which modern compilers t…

I wonder why there aren't more tools to allow structural editing like Lisp has. This way the file can never be in an unparsable state.

because in most languages there's a level of abstraction between the syntax of the language and the actual data structure of the program. To use the much maligned term, most languages aren't homoiconic, the internal structure of the program is hidden away from the programmer.

Lisps essentially have no syntax, that's why it is trivial to manipulate a lisp program structurally.

Post reply on HN