Hey, Tree-sitter author here. Thanks for posting! Let me know if you have questions about the project.
When I played around with tree sitter a bit I noticed there were situations where ast elements didn't exactly contain what I'd expect them to. For example: comments are represented in the AST but unfortunately they don't have the contents of the comment parsed out following the laguanges conventions.
I was wondering if this is a case I could open an issue about? Is this for the main tree sitter repo or should I open one language-by-language?
I was looking into automating some stuff across all languages with tree-sitter but handling all of the languages comments syntaxes made it very hard.
While we're in this discussion: Say I want to implement "SQL" for my app (if you've used Jira, I want to make my own JQL). Is this the tool for that? I'm looking for something much simpler than ANTLR.
Hey, Tree-sitter author here. Thanks for posting! Let me know if you have questions about the project.
When I played around with tree sitter a bit I noticed there were situations where ast elements didn't exactly contain what I'd expect them to. For example: comments are represented in the AST but unfortunately they don't have the contents of the comment parsed out following the laguanges conventions. I was wondering if this is a case I could open an issue about? Is this for the main tree sitter repo or should I open…
Most tree-sitter grammars just parse comments as a single token. Can you give an example of what you mean when you say "contents of the comment parsed out"?
Are you talking about conventions like JSDoc, for putting structured data inside of comments? On GitHub, we handle that by parsing JSDoc comments in a separate pass, using a separate parser. We do it this way because JSDoc isn't really part of the JavaScript language, not all projects use JSDoc, and not all applications are interested in parsing the text inside of comments.
Ah I see, so the reparation isn't avoidable for now? That doesn't seem very appropraite for compilers then.
Why would it not be appropriate? The only annoyance I see is that currently you will have to generate a good error message from it yourself, but a first pass at the problem shouldn't be too onerous.
Ok, I misunderstood. I thought it repaired without error sometimes but I see that you were clear that that isn't the case.
I tried to use this to ease the front end work load of students in a compiler project (building a C compiler) for a University course, so that the project could be focused on the more interesting middle and back end parts of the compiler.
However, reported bugs in the C grammar that saw no activity at all [1] made this impossible. From this small sample of experiences, I was left with the impression that Tree Sitter is great for things like syntax highlighting, where wrong results are annoying but not dramatic, but not so suitable for tools that need a really correct syntax tree.
Tree Sitter is amazing. The parsing is fast enough to run on every keystroke. The parse tree is extremely concise and readable. It resembles an AST more than a parse tree (ie no 11 levels of binary op precedence rules in the tree). The parse tree emits specific ERROR nodes, so you can get a semi-functional tree even with broken syntax. I can't wait for the tools to get built with this. Paredit for TypeScript. Syntax-…
A friend of mine started working on an experimental Emacs mode to provide structural navigation of code based on tree-sitter: https://cs.tvl.fyi/depot/-/tree/users/Profpatsch/emacs-tree-... The potential for this is essentially something like Paredit, but for all languages.
Can someone point some examples of what `paredit` for other languages provide? I do various lisp programming occasionally but have not used `paredit` yet.
A friend of mine started working on an experimental Emacs mode to provide structural navigation of code based on tree-sitter: https://cs.tvl.fyi/depot/-/tree/users/Profpatsch/emacs-tree-... The potential for this is essentially something like Paredit, but for all languages.
Can someone point some examples of what `paredit` for other languages provide? I do various lisp programming occasionally but have not used `paredit` yet.
Can someone point some examples of what `paredit` for other languages provide? I do various lisp programming occasionally but have not used `paredit` yet.
Check out this video for a quick demo: http://emacsrocks.com/e14.html If you know a Lisp I recommend just giving paredit a spin for a few minutes, it's an interesting experience.
Looks like it's mainly tree/code manipulation. Typing code on the keyboard is probably the least taxing thing when it comes to software development. But I guess it will be nice once it has become a "reflex" rather then a conscious key-combo.
Hey, Tree-sitter author here. Thanks for posting! Let me know if you have questions about the project.
There's been some recent discussion as to whether tree-sitter grammars can be used to parse markdown with some hacks or not (currently it's being done by working around all the tree-sitter machinery, resulting in a lot of problems), with no consensus among plugin authors:
Not a ruby developer here: that sounds terrifying! Does it make it harder to have a proper mental model of the language (note: not the libraries) or is this mainly because of flexibility (too many ways to skin one cat)?
I don't write Ruby regularly either, but I wouldn't say that syntactic complexity, is necessarily equivalent to semantic complexity. And the syntax is the only part that's relevant to Tree-sitter: it's not an interpreter/compiler. Note also that (as I alluded to above) the parsing technique that Tree-sitter uses, "LR parsing", makes some things more difficult to parse than they'd be with another kind of parser. This…
So, a syntactic tree is a list of elements, grouped by their ordering, which are to be parsed from their arguments, as they appeared in the input. Or a grammar tree, which is a set of elements.
There's many things we can do to make Tree-sitter simpler to read and write. Perhaps, like in Perl, there are syntactic categories of types that make it much easier to find things like nodes in a tree, since they're the ones that come in the input. Or I'd be willing to say that maybe, like in Haskell, certain aspects of the language, are syntactic categories, like the parser. So some things that might not be obvious in code, like what the syntax for a class of names is, might be obvious in theory, too. Or, at least they might be obvious in a particular way. Or some aspects of the compiler are really special, and we can infer those in terms of what the compiler does.
Or, of course, we can do all these other things, too. We can rewrite the parser, or the compiler, to try to do more or less anything that the parser does. Or maybe we can make Tree-sitter a lot simpler in general. Which I think is probably what you've been thinking about.