Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tree-sitter: an incremental parsing system for programming tools

github.com

Post reply on HN
Print view View original

81–90 of 138 posts

Re: Tree-sitter: an incremental parsing system for programming tools

#81

Post by eins1234 »

dcreager wrote: post #29
srcreigh wrote: post #11

Tree Sitter is amazing. The parsing is fast enough to run on every keystroke. The parse tree is extremely concise and readable. It resembles an AST more than a parse tree (ie no 11 levels of binary op precedence rules in the tree). The parse tree emits specific ERROR nodes, so you can get a semi-functional tree even with broken syntax. I can't wait for the tools to get built with this. Paredit for TypeScript. Syntax-…

Worth calling out that the syntax highlighting support is used to highlight several languages in github.com. (Linguist is still used for the long tail of languages, but we plan to migrate more and more over to tree-sitter-based highlighting over time.) The query language is also what's used to drive the fuzzy/ctags-like Code Navigation feature. Both of those are powered by tree-sitter query files defined in each lang…

Awesome to hear that amazing tech like tree-sitter lives on even though Atom, the product it was built for, is pretty much on life support at this point.

Curious if there's any efforts to bring tree-sitter to VSCode? Exposing tree-sitter to extensions could open up so many possibilities like OP mentioned.

Re: Tree-sitter: an incremental parsing system for programming tools

#82

Post by maxbrunsfeld »

dang wrote: post #80

If curious, past threads: Tree-sitter: new incremental parsing system for programming tools (2018) [video] - https://news.ycombinator.com/item?id=21675113 - Dec 2019 (28 comments) Tree-sitter – a new parsing system for programming tools [video] - https://news.ycombinator.com/item?id=18213022 - Oct 2018 (25 comments) Others?

One more that I know of:

Atom understands your code better than ever before - https://news.ycombinator.com/item?id=18349013 - Oct 2018

Re: Tree-sitter: an incremental parsing system for programming tools

#83

Post by josteink »

srcreigh wrote: post #11

Tree Sitter is amazing. The parsing is fast enough to run on every keystroke. The parse tree is extremely concise and readable. It resembles an AST more than a parse tree (ie no 11 levels of binary op precedence rules in the tree). The parse tree emits specific ERROR nodes, so you can get a semi-functional tree even with broken syntax. I can't wait for the tools to get built with this. Paredit for TypeScript. Syntax-…

Tooting my own horn, Emacs’ csharp-mode[1] is undergoing a rewrite to be 100% based on tree-sitter rather than regexps.

The new code runs way faster and is so much nicer to work with.

Once all the kinks are gone, I can’t imagine going back.

[1] https://github.com/emacs-csharp/csharp-mode/blob/master/csha...

Re: Tree-sitter: an incremental parsing system for programming tools

#84

Post by brundolf »

lalaithion wrote: post #27
srcreigh wrote: post #11

Tree Sitter is amazing. The parsing is fast enough to run on every keystroke. The parse tree is extremely concise and readable. It resembles an AST more than a parse tree (ie no 11 levels of binary op precedence rules in the tree). The parse tree emits specific ERROR nodes, so you can get a semi-functional tree even with broken syntax. I can't wait for the tools to get built with this. Paredit for TypeScript. Syntax-…

Maybe I can finally have this syntax highlighting style: https://youtu.be/b0EF0VTs9Dc?t=900

The idea is pretty awesome, but my eyes nearly rolled out of my head from the needless condescension at the beginning.

Re: Tree-sitter: an incremental parsing system for programming tools

#85

Post by josephg »

maxbrunsfeld wrote: post #28

Hey, Tree-sitter author here. Thanks for posting! Let me know if you have questions about the project.

There's an architecture for compilers that I've been wanting for years where a keystroke change to the sourcecode results in an incremental change to the AST, and then the compiler can consume that AST delta to generate a binary patch to the compiled executable.

Would tree-sitter be able to be used for that? (What I want is to feed tree-sitter a stream of keystroke changes and get out a stream of minimal AST changes as a result).

Re: Tree-sitter: an incremental parsing system for programming tools

#86

Post by gugagore »

maxbrunsfeld wrote: post #63
gravypod wrote: post #61

Earlier quoted context omitted.

When I played around with tree sitter a bit I noticed there were situations where ast elements didn't exactly contain what I'd expect them to. For example: comments are represented in the AST but unfortunately they don't have the contents of the comment parsed out following the laguanges conventions. I was wondering if this is a case I could open an issue about? Is this for the main tree sitter repo or should I open…

Most tree-sitter grammars just parse comments as a single token. Can you give an example of what you mean when you say "contents of the comment parsed out"? Are you talking about conventions like JSDoc, for putting structured data inside of comments? On GitHub, we handle that by parsing JSDoc comments in a separate pass, using a separate parser. We do it this way because JSDoc isn't really part of the JavaScript lang…

My guess is that they meant parsing code that has been "commented out".

Re: Tree-sitter: an incremental parsing system for programming tools

#87

Post by mumblemumble »

z3t4 wrote: post #68
tazjin wrote: post #67

Earlier quoted context omitted.

Check out this video for a quick demo: http://emacsrocks.com/e14.html If you know a Lisp I recommend just giving paredit a spin for a few minutes, it's an interesting experience.

Looks like it's mainly tree/code manipulation. Typing code on the keyboard is probably the least taxing thing when it comes to software development. But I guess it will be nice once it has become a "reflex" rather then a conscious key-combo.

I only started using it a few months ago. It's such a natural way to edit code, it only took me about a day for it to become reflexive.

Now it just feels vaguely annoying to work without it. It's fine, it's just one of those ergonomic changes that nags at you a bit. Kind of like the opposite of that feeling of taking off uncomfortable business clothes at the end of the day. Or what I imagine people who are better at vim than me keep talking about.

Re: Tree-sitter: an incremental parsing system for programming tools

#88

Post by chrisseaton »

tp3 wrote: post #71
chrisseaton wrote: post #33

Earlier quoted context omitted.

I didn't mean the tree-sitter grammar was not understandable - it's very understandable - I just can't work out how to managed to find such a concise way to express grammars. Even compared to Whitequark it's 1/3 the size. What's the unique thing you do that makes it so concise? It also seems somehow to be completely declarative? How have you managed to transform Ruby parsing to be context-free? For example where's th…

The code is obviously much simpler than its syntax - most importantly, its syntactical simplicity makes it way easier to deal with. So when you write the code to parse it you don't have to try to parse it in one fell swoop like you do in Whitequark. So you can't read anything from a method call! I can make it so, if you're doing a class method (of any kind) you have to invoke the constructor, as described in "What is…

> The code is obviously much simpler than its syntax

What code? The parser? How can it be simpler than its syntax? It has syntax and semantics, which is strictly more than the syntax.

> The point I'm making here is that LR doesn't give a reason for what you're doing.

What do you mean 'what you're doing'?

Re: Tree-sitter: an incremental parsing system for programming tools

#89

Post by efritz »

I'm an engineer on the code intelligence team at Sourcegraph.

We've been busy building out true precise code intelligence/navigation support, but we also have a mode for zero-configuration code navigation based on text search, universal-ctags, and hand-rolled regular expressions (which works surprisingly well!). Tree-sitter would definitely give better results than our current ctags-based approach. It's been catching our attention more and more lately, and we have plans to use it to upgrade our out-of-the-box, instant code navigation experience.

It's not the exact right fit for our primary goals though, since it's designed around being extremely fast while editing and robust against errors. Sourcegraph is only used for navigating committed code, so we're leveraging formats like LSIF to generate complete semantic graphs of codebases and their entire dependency tree. That'll enable a lot of features that are out of reach for tree-sitter, but is a lot harder to get working out of the box and it's a much bigger technical investment.

It's very interesting to see the topological space that houses these solutions fill out. Every tool has its own set of unique trade-offs and fall somewhere on these spectrums:

- fast vs slow

- precise vs imprecise

- zero-configuration vs configuration required

We've visited a few islands in this space but still very curious to see what other islands can be discovered. We're especially excited about tools and formats like tree-sitter and LSIF around which a large and supportive community can grow so that all the products we love and rely on as developers can all make forward progress.

Re: Tree-sitter: an incremental parsing system for programming tools

#90

Post by rattray »

gugagore wrote: post #86
maxbrunsfeld wrote: post #63

Earlier quoted context omitted.

Most tree-sitter grammars just parse comments as a single token. Can you give an example of what you mean when you say "contents of the comment parsed out"? Are you talking about conventions like JSDoc, for putting structured data inside of comments? On GitHub, we handle that by parsing JSDoc comments in a separate pass, using a separate parser. We do it this way because JSDoc isn't really part of the JavaScript lang…

My guess is that they meant parsing code that has been "commented out".

I interpreted it to mean, "Remove the *s from code like this:"

    /* This comment
     * Should just be alphanumeric.
     */
Post reply on HN