Always kudos towards taking a self-hosted-forge approach
Tbsp – treesitter-based source processing language
11–20 of 47 posts
Re: Tbsp – treesitter-based source processing language
#12Maybe update the link to https://git.peppe.rs/languages/tbsp/tree/readme.txt ?
Is it formerly peppe.rs ?
Here is the new account and doc for tbsp below.
Re: Tbsp – treesitter-based source processing language
#13This is great, and a step in the right direction. I wish tree-sitter had an official higher level API that allowed processing and pattern matching for use cases other than those required for text editors.
I’m currently using tree-sitter at work to build AST-based tools, as performance is amazing, even with huge codebases, but I’m finding it slightly frustrating to have to manually write recursive descent processors keyed by strings, with no compile time guarantees on the structure of the grammar.
This is compounded by the fact that grammars themselves don’t really follow any standard structure, some have named fields (presumably the ones created after GitHub contributed this feature), while others require hierarchical pattern matching.
I wish there existed a tool to consume a grammar and output a rust ADT that we can simply match on. This would at least save me from redundant error handling. I’d build one myself, but I’m that good at rust yet.
Re: Tbsp – treesitter-based source processing language
#14This is so cool. Question (caveat: first export to treesitter and tools like this): Is there a reason the example demonstrates the use of depth as a variable instead of it being built in? Nesting level of a particular "type" is general enough that it might be included OOTB. What you want to do with this might be generalizable - for example instead of ``` enter section { depth += 1; } leave section { depth -= 1; } ent…
The depth here can be context dependent. For example if you had a bunch of brackets and parens in your grammar, you might only care about paren depth. Or if your language had brackets and parens and function definitions, your "expression depth" might ignore function definitions (or even reset at a function definition boundary if you have inner functions!)
Re: Tbsp – treesitter-based source processing language
#15The md-to-html demo is a good one, but worth mentioning that the Markdown parser[1] being used may not be suitable for more complex documents. From the README:
> "...it is not recommended to use this parser where correctness is important. The main goal for this parser is to provide syntactical information for syntax highlighting..."
There's also a separate block-level and inline parser, not sure how `tbsp` handles nested or multi-stage parsing.
[1]: https://github.com/tree-sitter-grammars/tree-sitter-markdown
Re: Tbsp – treesitter-based source processing language
#16This is great, and a step in the right direction. I wish tree-sitter had an official higher level API that allowed processing and pattern matching for use cases other than those required for text editors. I’m currently using tree-sitter at work to build AST-based tools, as performance is amazing, even with huge codebases, but I’m finding it slightly frustrating to have to manually write recursive descent processors k…
I’ve also encountered this problem using various tree-sitter grammars. I would love a data set that showed various implementations for different languages, along with some kind of consistent test coverage for each language that shows compatibility versus the compiler’s parser. And, of course, links to precompiled wasm modules. Basically, a tree-sitter package manager.
Re: Tbsp – treesitter-based source processing language
#17Adding a way to query the path at the current node would let you skip out on doing stuff like keeping track of `in_section`.
I wonder if the `enter|exit ...` syntax might be too limiting but for a lot of stuff it seems nice and easy to reason about. Easier than tree-sitter's own queries.
I think if you really wanted performance and whatnot, you might end up compiling the queries to another target and just reuse them.
I could see myself writing a lua DSL around compiling these kinds of queries `enter/exit` stanzas or an SQL one too.
Re: Tbsp – treesitter-based source processing language
#18This is great, and a step in the right direction. I wish tree-sitter had an official higher level API that allowed processing and pattern matching for use cases other than those required for text editors. I’m currently using tree-sitter at work to build AST-based tools, as performance is amazing, even with huge codebases, but I’m finding it slightly frustrating to have to manually write recursive descent processors k…
> I wish tree-sitter had an official higher level API that allowed processing and pattern matching for use cases other than those required for text editors.
Is the pattern matching API not sufficiently high level? In my experience, it's a huge improvement over implementing visitors for everything.
https://tree-sitter.github.io/tree-sitter/using-parsers#patt...
Re: Tbsp – treesitter-based source processing language
#19Awesome! I'd love to see this flourish.
Re: Tbsp – treesitter-based source processing language
#20For those that want to explore the grammars listed at https://github.com/tree-sitter/tree-sitter/wiki/List-of-pars... in a more friendly railroad diagram format I made https://mingodad.github.io/plgh/json2ebnf.html that reads the "src/grammar.json" and try it's best to generate an EBNF understood by (IPV6) https://www.bottlecaps.de/rr/ui or (IPV4) https://rr.red-dove.com/ui where we get a nice navigable railroad diagram (see https://github.com/GuntherRademacher/rr for offline usage).