Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tree-sitter: an incremental parsing system for programming tools

github.com

Post reply on HN
Print view View original

51–60 of 138 posts

Re: Tree-sitter: an incremental parsing system for programming tools

#51

Post by tazjin »

srcreigh wrote: post #11

Tree Sitter is amazing. The parsing is fast enough to run on every keystroke. The parse tree is extremely concise and readable. It resembles an AST more than a parse tree (ie no 11 levels of binary op precedence rules in the tree). The parse tree emits specific ERROR nodes, so you can get a semi-functional tree even with broken syntax. I can't wait for the tools to get built with this. Paredit for TypeScript. Syntax-…

A friend of mine started working on an experimental Emacs mode to provide structural navigation of code based on tree-sitter: https://cs.tvl.fyi/depot/-/tree/users/Profpatsch/emacs-tree-...

The potential for this is essentially something like Paredit, but for all languages.

Re: Tree-sitter: an incremental parsing system for programming tools

#52

Post by maxbrunsfeld »

fiddlerwoaroof wrote: post #50
maxbrunsfeld wrote: post #28

Hey, Tree-sitter author here. Thanks for posting! Let me know if you have questions about the project.

I’ve been using tree-sitter via FFI from Common Lisp, but what I’d really like would be a way to write my own code generator so that the generated parser could be “native” lisp code. Otherwise, it’s an amazing tool: my only other complaint would be the lack of a grammar for objective-c which would be useful for a lisp/objective-c bridge I’ve been working on.

I think that it'd be pretty easy to generate parser code in other languages besides C, but it would be a lot of work to do to port the core library itself[1] to those other languages.

[1] https://github.com/tree-sitter/tree-sitter/tree/master/lib/s...

I agree about the Objective-C grammar! Although it looks like somebody's started work on it:

https://github.com/merico-dev/tree-sitter-objc

Re: Tree-sitter: an incremental parsing system for programming tools

#53

Post by jackcviers3 »

srcreigh wrote: post #49
lalaithion wrote: post #27

Earlier quoted context omitted.

Maybe I can finally have this syntax highlighting style: https://youtu.be/b0EF0VTs9Dc?t=900

There is an emacs package for this (maybe beta). I can't remember the name of it and Google is failing me. EDIT: finally found it https://github.com/alphapapa/prism.el

Rainbow delimiters mode kind of does this, but doesn't maintain the scope color of referenced variables.

Re: Tree-sitter: an incremental parsing system for programming tools

#54

Post by sitkack »

dcreager wrote: post #37
yig wrote: post #34

Earlier quoted context omitted.

Are there any plans to support modifying the grammar on the fly or without recompiling?

I don't think you can do this without recompiling, since the grammars get translated into C code before use. But the built-in command line tools (‘tree-sitter parse’, etc) all support a mode where they will detect local changes to a checked-out grammar definition, and recompile on the fly if needed. (This happens each time the CLI program is started up; it doesn't happen during a long-running process.)

The obvious answer is to embed TCC or another C compiler and either generate a dynamic library or generate wasm and load it directly into the process.

exec_wasm(generate_wasm(generate_c(grammar)))

Now if you can make that whole fn chain incremental, then a delta_grammar -> delta_c -> delta_wasm -> delta_recomputed_wasm_call stack, this will propagate deltas down to exec_wasm and you could dynamically execute the generated code as the grammar changes.

Re: Tree-sitter: an incremental parsing system for programming tools

#55

Post by dcreager »

brundolf wrote: post #48
dcreager wrote: post #35

Earlier quoted context omitted.

We also have several of the language grammars published as crates: https://crates.io/search?q=tree-sitter (And doing the same for other grammars is a fairly painless process.) So if you're writing a tool for a single language (like a language server), it should be as easy as adding tree-sitter and tree-sitter-blah to your cargo manifest.

Awesome! Though my thinking was that it would have an especially large impact for languages that aren't popular enough to have their own LSP yet; you no longer have to be an expert in writing interactive compilers to set up a respectable LSP for a niche language, or even a home-grown one

Yes! This is a great point. It's similar to what I mentioned over on this thread [1] about how we're working on a more precise version of Code Navigation based on tree-sitter. The tl;dr is that you'd write something like tree-sitter queries [2], just like you do for the current fuzzy Code Nav, but the query DSL would be a bit more sophisticated, allowing you to specify the actual name resolution rules of your language. One of the things we're using to test this is an LSP shim that lets us test our rules in VS Code (or any other LSP-compliant editor).

[1] https://news.ycombinator.com/item?id=26227476 [2] https://tree-sitter.github.io/tree-sitter/using-parsers#patt...

Re: Tree-sitter: an incremental parsing system for programming tools

#56

Post by robto »

srcreigh wrote: post #11

Tree Sitter is amazing. The parsing is fast enough to run on every keystroke. The parse tree is extremely concise and readable. It resembles an AST more than a parse tree (ie no 11 levels of binary op precedence rules in the tree). The parse tree emits specific ERROR nodes, so you can get a semi-functional tree even with broken syntax. I can't wait for the tools to get built with this. Paredit for TypeScript. Syntax-…

I'm so excited for this to become built-in in more places! I think once non-lisp users can experience the Power of Structural Editing they'll say, "Hey, I understand now why you all feel so passionate about your parentheses!"

And I can stop feeling like my fingers have all lost a knuckle when I'm writing Typescript :)

Re: Tree-sitter: an incremental parsing system for programming tools

#57

Post by mkingston »

rgossiaux wrote: post #42
srcreigh wrote: post #11

Tree Sitter is amazing. The parsing is fast enough to run on every keystroke. The parse tree is extremely concise and readable. It resembles an AST more than a parse tree (ie no 11 levels of binary op precedence rules in the tree). The parse tree emits specific ERROR nodes, so you can get a semi-functional tree even with broken syntax. I can't wait for the tools to get built with this. Paredit for TypeScript. Syntax-…

Neovim nightly already has some tools available as plugins. I'm using tree-sitter for syntax highlighting, text objects, and folding right now. Pretty satisfied so far.

The official release of built-in treesitter comes with neovim 0.5. Which looks like it'll be out pretty soon. I've been watching a fairly steady march toward release here: https://github.com/neovim/neovim/milestone/19

Re: Tree-sitter: an incremental parsing system for programming tools

#58

Post by e12e »

ComputerGuru wrote: post #17
anaerobicover wrote: post #15

Earlier quoted context omitted.

No, the Ruby grammar is actually an outlier from what I've seen; it has one of the largest/most complex external scanners: https://github.com/tree-sitter/tree-sitter-ruby/blob/master/... Precisely because the language is complicated and less amenable to LR parsing.

Not a ruby developer here: that sounds terrifying! Does it make it harder to have a proper mental model of the language (note: not the libraries) or is this mainly because of flexibility (too many ways to skin one cat)?

> Not a ruby developer here: that sounds terrifying! Does it make it harder to have a proper mental model of the language

It is a little terrifying in the sense that I'd not want to write language level tools (eg: syntax highlighter).

But if you have scheme on one end and natural language on the other, ruby leans à bit towards natural language - but in a good way. In some ways ruby isn't that different from Smalltalk - but it has a lot (sometimes I think too many, sometimes not) conveniences.

Parantheses and brackets are largely optional "where it makes sense". Conditionals support postfix, eg these are equivalent:

  if should_send?() 
    send_mail({to: 'u@x.com'}) 
  end

  send_mail to: 'u@x.com' if should_send?

Re: Tree-sitter: an incremental parsing system for programming tools

#59

Post by mkingston »

ckolkey wrote: post #3
ducktective wrote: post #2

Is this the same thing neovim uses for syntax highlighting? Is there a chance for it getting integrated to vim? Last I checked vim used a regex method which was slow and faulty.

Yup, neovim 0.5+ will be using treesitter for any supported languages, with the current Regex highlighting as a fallback.

Follow the nvim 0.5 release here: https://github.com/neovim/neovim/milestone/19
Post reply on HN