Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tree-sitter: an incremental parsing system for programming tools

github.com

Post reply on HN
Print view View original

111–120 of 138 posts

Re: Tree-sitter: an incremental parsing system for programming tools

#111

Post by tclem »

chrisseaton wrote: post #20
codesnik wrote: post #19

Earlier quoted context omitted.

bison should be compared to https://github.com/tree-sitter/tree-sitter-ruby/blob/master/... probably?

No the JSON file there is generated (I believe?) from the JavaScript I linked, while the Bison file is hand-written. With tree-sitter you're hand-writing a 1k file. With Bison you're hand-writing a 13k file.

@chrisseaton you are correct, the JSON file is generated. The handwritten parts are:

- https://github.com/tree-sitter/tree-sitter-ruby/blob/32cd5a0... - https://github.com/tree-sitter/tree-sitter-ruby/blob/32cd5a0...

So about 2k loc.

The trickiest (and most verbose) parts of the external scanner have to do with heredocs and the various ways to declare literals (strings, symbols, regexes, etc).

Re: Tree-sitter: an incremental parsing system for programming tools

#112

Post by ubolonton_ »

dcreager wrote: post #55
brundolf wrote: post #48

Earlier quoted context omitted.

Awesome! Though my thinking was that it would have an especially large impact for languages that aren't popular enough to have their own LSP yet; you no longer have to be an expert in writing interactive compilers to set up a respectable LSP for a niche language, or even a home-grown one

Yes! This is a great point. It's similar to what I mentioned over on this thread [1] about how we're working on a more precise version of Code Navigation based on tree-sitter. The tl;dr is that you'd write something like tree-sitter queries [2], just like you do for the current fuzzy Code Nav, but the query DSL would be a bit more sophisticated, allowing you to specify the actual name resolution rules of your languag…

> the query DSL would be a bit more sophisticated, allowing you to specify the actual name resolution rules of your language.

This sounds very interesting. Will the query DSL (spec) be available to the public?

Re: Tree-sitter: an incremental parsing system for programming tools

#113

Post by nerdponx »

lalaithion wrote: post #27
srcreigh wrote: post #11

Tree Sitter is amazing. The parsing is fast enough to run on every keystroke. The parse tree is extremely concise and readable. It resembles an AST more than a parse tree (ie no 11 levels of binary op precedence rules in the tree). The parse tree emits specific ERROR nodes, so you can get a semi-functional tree even with broken syntax. I can't wait for the tools to get built with this. Paredit for TypeScript. Syntax-…

Maybe I can finally have this syntax highlighting style: https://youtu.be/b0EF0VTs9Dc?t=900

I'm pretty sure you can implement this in Neovim with Tree Sitter now.

This would be a wonderful idea in any programming language.

But I agree with the other commenter that this speaker is really condescending. Not a person I'd want to work with.

Re: Tree-sitter: an incremental parsing system for programming tools

#114

Post by patrec »

efritz wrote: post #89

I'm an engineer on the code intelligence team at Sourcegraph. We've been busy building out true precise code intelligence/navigation support, but we also have a mode for zero-configuration code navigation based on text search, universal-ctags, and hand-rolled regular expressions (which works surprisingly well!). Tree-sitter would definitely give better results than our current ctags-based approach. It's been catching…

What are those features out of reach of tree-sitter? I can see that you theoretically want something that's optimized for parsing well-formed code all at once, rather than potentially malformed code incrementally, but what trade-offs does tree-sitter make in practice that limit its potential for your use case? On the face of it, it seems to me like tree-sitter could server as a perfectly fine building block for generating LSIF or whatever from a code file.

Re: Tree-sitter: an incremental parsing system for programming tools

#115

Post by komodo009 »

nerdponx wrote: post #113
lalaithion wrote: post #27

Earlier quoted context omitted.

Maybe I can finally have this syntax highlighting style: https://youtu.be/b0EF0VTs9Dc?t=900

I'm pretty sure you can implement this in Neovim with Tree Sitter now. This would be a wonderful idea in any programming language. But I agree with the other commenter that this speaker is really condescending. Not a person I'd want to work with.

nvim-treesitter is pretty promising, do you know if it supports this type of highlighting already?

Re: Tree-sitter: an incremental parsing system for programming tools

#116

Post by throwaway_2047 »

tazjin wrote: post #67
yewenjie wrote: post #66

Earlier quoted context omitted.

Can someone point some examples of what `paredit` for other languages provide? I do various lisp programming occasionally but have not used `paredit` yet.

Check out this video for a quick demo: http://emacsrocks.com/e14.html If you know a Lisp I recommend just giving paredit a spin for a few minutes, it's an interesting experience.

Can someone recommend what's the best material to learn emacs and lisp at the same time

Re: Tree-sitter: an incremental parsing system for programming tools

#117

Post by chewxy »

brundolf wrote: post #84
lalaithion wrote: post #27

Earlier quoted context omitted.

Maybe I can finally have this syntax highlighting style: https://youtu.be/b0EF0VTs9Dc?t=900

The idea is pretty awesome, but my eyes nearly rolled out of my head from the needless condescension at the beginning.

it's just how old people talk. Rob Pike speaks of syntax colouring in the same way - he quotes the Bible.

I don't see it as condescension though. I think it's just a way of speaking.

Re: Tree-sitter: an incremental parsing system for programming tools

#119

Post by efritz »

patrec wrote: post #114
efritz wrote: post #89

I'm an engineer on the code intelligence team at Sourcegraph. We've been busy building out true precise code intelligence/navigation support, but we also have a mode for zero-configuration code navigation based on text search, universal-ctags, and hand-rolled regular expressions (which works surprisingly well!). Tree-sitter would definitely give better results than our current ctags-based approach. It's been catching…

What are those features out of reach of tree-sitter? I can see that you theoretically want something that's optimized for parsing well-formed code all at once, rather than potentially malformed code incrementally, but what trade-offs does tree-sitter make in practice that limit its potential for your use case? On the face of it, it seems to me like tree-sitter could server as a perfectly fine building block for gener…

> On the face of it, it seems to me like tree-sitter could server as a perfectly fine building block for generating LSIF or whatever from a code file.

It does seem this way. Another reply [1] this post makes the same point with a nice proof-of-concept as well.

[1]: https://news.ycombinator.com/item?id=26230900

Re: Tree-sitter: an incremental parsing system for programming tools

#120

Post by dcreager »

ubolonton_ wrote: post #112
dcreager wrote: post #55

Earlier quoted context omitted.

Yes! This is a great point. It's similar to what I mentioned over on this thread [1] about how we're working on a more precise version of Code Navigation based on tree-sitter. The tl;dr is that you'd write something like tree-sitter queries [2], just like you do for the current fuzzy Code Nav, but the query DSL would be a bit more sophisticated, allowing you to specify the actual name resolution rules of your languag…

> the query DSL would be a bit more sophisticated, allowing you to specify the actual name resolution rules of your language. This sounds very interesting. Will the query DSL (spec) be available to the public?

That's the current plan! In particular, because we want to allow language communities to implement support for their own languages, and not have to be blocked on my team finding the time to do it. (Just like they can do now with the parser and syntax highlighting / fuzzy code nav rules.) Linguist is our role model here — it currently includes language detection and (regex-based) syntax highlighting rules for 500+ languages. Most of those are contributed by the community. There's no way that my team can migrate all of those in any reasonable amount of time, especially while having to balance that with other feature development and operational responsibilities.
Post reply on HN