Speeding up VSCode extensions in 2022
jason-williams.co.uk
Speeding up VSCode extensions in 2022
1–10 of 40 posts
Re: Speeding up VSCode extensions in 2022
#2The tokenization speed issue is already addressed by the language server protocol, which delegates syntax highlighting to language servers via the "semantic tokens" API. Language servers can choose to implement this however they want, and the API allows for incremental additions.
I think this is a much better approach that baking tree sitter into VS Code and continuing to use TextMate grammars (or tree sitter specific grammars) to add syntax highlighting. That way it can be applied to any editor that supports LSP integration. Really the world would be a better place for IDEs and text editors if we could just get LSP more standardized, and VS Code's dominance is a decent place to start with it. I'm tired of relying on editor and IDE authors for language support.
Re: Speeding up VSCode extensions in 2022
#3The tokenization speed issue is already addressed by the language server protocol, which delegates syntax highlighting to language servers via the "semantic tokens" API. Language servers can choose to implement this however they want, and the API allows for incremental additions. I think this is a much better approach that baking tree sitter into VS Code and continuing to use TextMate grammars (or tree sitter specifi…
I believe I've read in various places that the difficulty of doing syntax highlighting fast enough is precisely why it isn't done via LSP.
Am I right in thinking that the Semantic Tokens part of the spec (currently) falls short of saying "this is for all your syntax-highlighting needs, please go ahead and implement full-blown syntax highlighters using it"
I definitely agree with you that the more we can share between different editors/IDEs the better.
Re: Speeding up VSCode extensions in 2022
#4The tokenization speed issue is already addressed by the language server protocol, which delegates syntax highlighting to language servers via the "semantic tokens" API. Language servers can choose to implement this however they want, and the API allows for incremental additions. I think this is a much better approach that baking tree sitter into VS Code and continuing to use TextMate grammars (or tree sitter specifi…
> The tokenization speed issue is already addressed by the language server protocol [...]
It's not really addressed. The semantic tokens API is intended for semantic highlighting:
> Semantic tokenization allows language servers to provide additional token information based on the language server's knowledge on how to resolve symbols in the context of a project.
Abusing the semantic tokens API for syntactic highlighting is slow, unnecessarily complex (why do I need to implement a language server just to do syntactic highlighting?), and only a partial solution (still need a TM grammar, still don't get correct code folding, etc.).
Re: Speeding up VSCode extensions in 2022
#5The tokenization speed issue is already addressed by the language server protocol, which delegates syntax highlighting to language servers via the "semantic tokens" API. Language servers can choose to implement this however they want, and the API allows for incremental additions. I think this is a much better approach that baking tree sitter into VS Code and continuing to use TextMate grammars (or tree sitter specifi…
> I think this is a much better approach that baking tree sitter into VS Code
they're implementing both, with tree sitter being 'dumb' version of LSP syntax highlighting: https://github.com/microsoft/vscode-anycode
Re: Speeding up VSCode extensions in 2022
#6The tokenization speed issue is already addressed by the language server protocol, which delegates syntax highlighting to language servers via the "semantic tokens" API. Language servers can choose to implement this however they want, and the API allows for incremental additions. I think this is a much better approach that baking tree sitter into VS Code and continuing to use TextMate grammars (or tree sitter specifi…
> I think this is a much better approach that baking tree sitter into VS Code they're implementing both, with tree sitter being 'dumb' version of LSP syntax highlighting: https://github.com/microsoft/vscode-anycode
Oh thanks for that link, that's really interesting.
Can someone explain the paragraph below -- I thought "this is an invocation of a function named bar" was what we mean by semantic information:
> All features are based on parse trees and there is no semantic information - that means there is no guarantee for correctness. Parse trees allow to identify declarations and usages, like "these lines define a function named foo" or "this is an invocation of a function named bar"
Re: Speeding up VSCode extensions in 2022
#7Can we stop with the "in 2022" headlines?
Re: Speeding up VSCode extensions in 2022
#8Can we stop with the "in 2022" headlines?
It's an SEO thing.
Re: Speeding up VSCode extensions in 2022
#9The tokenization speed issue is already addressed by the language server protocol, which delegates syntax highlighting to language servers via the "semantic tokens" API. Language servers can choose to implement this however they want, and the API allows for incremental additions. I think this is a much better approach that baking tree sitter into VS Code and continuing to use TextMate grammars (or tree sitter specifi…
> I think this is a much better approach that baking tree sitter into VS Code they're implementing both, with tree sitter being 'dumb' version of LSP syntax highlighting: https://github.com/microsoft/vscode-anycode
What's "dumb" about Anycode is its semantic features such as "go to definition", code completion, etc., not its syntax highlighting. Alas, Anycode is a separate project.
Re: Speeding up VSCode extensions in 2022
#10Speaking of tree-sitter, I’ve been experimenting with it for a static analysis and codegen idea and… wow it is fast. And super easy to use. I threw together a naive TypeScript type-stripping “compiler”, using the Node bindings so it’s got a couple bottlenecks. It’s within spitting distance of esbuild for a huge (10k loc) real world module (about 80-90ms vs ESBuild’s 20-30ms), and sometimes faster than esbuild for small (50-100 loc) modules. Granted it’s not doing everything esbuild does. But it was a quick experiment with a familiar domain before I go further. Quick as in mostly working within a few hours, and naively optimized for large source content in another hour.
The WASM bindings also perform very well (better in some cases), so it can be used anywhere WASM can. Which, it seems to me, means it’s a very good candidate to replace Babel without depending on language-specific tooling like SWC/Rome/Bun.