Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Optimizations in Syntax Highlighting

code.visualstudio.com

Post reply on HN
Print view View original

41–50 of 55 posts

Re: Optimizations in Syntax Highlighting

#41

Post by atombender »

alexdima wrote: post #37
iveqy wrote: post #24

So VSCode is great in many ways, and the article might be interesting. But I would never call it fast. It's still really really slow. Just see this comparision: https://www.youtube.com/watch?v=nDRBxtEUOFE

When we started the project, we did write tokenizers by hand. I mention that in the blog post. You can write some very fast tokenizers by hand, even in JavaScript. Of course they won't be as fast as hand written tokenizers in C, but you'd be surprised how well the code of a hand written tokenizer in JavaScript can be optimized by a JS engine, at least I was :). IR Hydra 2 is a great tool to visualise v8's IR represen…

What about a parser generator that takes something like a BNF-type language and generates optimal JS/TS code on the fly, similar to Lex/Yacc? (The BNF would be portable, the generated code would be a cache.)

I can see that reusing .tmLanguage files saves a lot of work, but that format is atrocious -- hard to both read and write. (I once wrote a parser/highlighter for it in ObjC, it was not a lot of fun.)

Re: Optimizations in Syntax Highlighting

#42

Post by okdana »

jbmorgado wrote: post #38
trishume wrote: post #33

Earlier quoted context omitted.

Copy-pasted from a comment I made elsewhere: It's harder than it sounds if you want to support many languages. The Sublime syntaxes repo I use has 34,000 lines of grammars whereas my engine is only 3000 lines of code. If you count all the tmLanguage files for nice languages available online it's probably hundreds of thousands of lines, and that's in a pretty dense format. The whole point of using tmLanguage files is…

Thank you, that explains a lot, but now a new doubt came from it: Why don't we use other text editors grammars that are simpler/quicker to parse in JS? I have no idea on the technicalities, but for instance, Vim or Emacs grammars instead?

I can't speak to Emacs, but i know that Vim's syntax definitions are (a) not as powerful as TM's, (b) a nightmare to maintain, and (c) heavily reliant on features specific to Vim's loony regular-expression engine (like variable-width look-around).

My experience is that most syntax highlighters and their definition formats (Scintilla, nano, almost anything that's based on JavaScript) are very limited/naïve compared to TextMate's. It doesn't have to be that way -- TM can certainly be improved upon -- but it is.

Re: Optimizations in Syntax Highlighting

#43

Post by nhaehnle »

alexdima wrote: post #31
nhaehnle wrote: post #23

Earlier quoted context omitted.

Thanks for this. Clearly the original post describes good work, but I can't help feeling the JS community is slacking off when it comes to performance. Just eyeballing the cited numbers, they take 3939ms to handle a 1.18MB input on "a somewhat powerful desktop machine". Assuming that that means a chip running at 2GHz, we're talking about over 6300 cycles per byte! That's quite frankly ridiculous. An improvement by at…

I hate slowness and inefficiency too, that's why I try to make the editor as fast as possible :), but at least in this case, it is not the dynamic nature of JS to blame, but rather the nature of TM grammars. TM grammars consist of rules that have regular expressions, which need to be constantly evaluated; and in order to implement a correct TM grammar interpreter, you must evaluate them. I've looked in the past for o…

Do you pre-process the regular expressions into a common DFA, or does oniguruma do that for you? That would seem like the natural design for this.

It's non-trivial because TextMate grammar seem like they're just a little bit too general to be convenient. So there's definitely a trade-off. But if I wanted to really get as fast as possible, I would try to see if I can get there.

Re: Optimizations in Syntax Highlighting

#45

Post by ex3ndr »

Does VS Code support highlighting for "non-regexp" cases. For example: in code i can reference a Class by it's name, but in the same time if could be a function - how you can distinct one from another by just regexp when you don't use capitalization marker for class names? Some times Class is an Object in some languages (Scala, Kotlin...), how this case is handled?

Re: Optimizations in Syntax Highlighting

#46

Post by bicubic »

joshschreuder wrote: post #44

Those who switched from ST to VS Code, did you stick with VS Code? Do you have any advice for making the transition easier - key bindings, packages etc.?

I stuck with VS Code. To be honest, I don't think it's even in the same league as ST or Atom due to the amount of integrated tooling. The integrated debugger, git, and task runner management is a god send. VS Code is also the only editor I've used where javascript type lookups and auto completion/code doc parsing Just Works out of the box.

Re: Optimizations in Syntax Highlighting

#47

Post by laurencerowe »

joshschreuder wrote: post #44

Those who switched from ST to VS Code, did you stick with VS Code? Do you have any advice for making the transition easier - key bindings, packages etc.?

I've switched over in the past couple of months. The motivating factor for me was my switch to writing more JS (the debugger works really well.) But I'm increasingly finding myself using it for everything I used to use Sublime for. It's come a long way in the past year or so since I last gave it a try.

Re: Optimizations in Syntax Highlighting

#48

Post by Groxx »

pygy_ wrote: post #40
Groxx wrote: post #28

Earlier quoted context omitted.

Is there a reason the oniguruma syntax can't be translated into other regexes? It'd require a fairly feature-rich engine since oniguruma supports practically everything, but is Chrome missing something fundamental that can't be mimicked? E.g. [::alpha::] could be converted to [a-zA-Z] or the UTF equivalent (though I doubt this particular one is a problem).

They have lookbehind lookup that's only coming now in ESNext , and, more importantly, recursion, if I understand properly[0]. Maybe more things, I stopped reading the docs at that point to comment here. They have other features absent from JS that can be more or less polyfilled, like possessive quantifiers and sticky matching (they do it with an escape sequence, though, so it can only be polyfilled using this trick[1…

Yeah, if lookbehinds aren't supported yet, that'd be death. Thanks!

Re: Optimizations in Syntax Highlighting

#49

Post by dgfgfdagasdfgfa »

trishume wrote: post #27
octref wrote: post #17

Earlier quoted context omitted.

Sorry but I'm not familiar with Rust -- is it possible to run your lib in all platforms in a Node environment? And, do you have an issue for the regex port? Would love to see the benchmark.

Yes it would be possible to write a C wrapper for my Rust library and link that from Node, however I expect you might lose a lot of performance to data conversion. I have yet to do the regex engine port, I'm doing that later this evening. I'm porting it from Oniguruma to https://github.com/google/fancy-regex which accelerates common types of regexes using the awesome Rust regex crate which is super fast.

Doesn't rust export the C abi? Or does the "c wrapper" just involve converting c types to rust domain types>

Re: Optimizations in Syntax Highlighting

#50

Post by hex13 »

flamedoge wrote: post #35
whatever_dude wrote: post #22

Earlier quoted context omitted.

A language service is for people with poor memory. A terminal window is what turns a text editor into a code editor.

I need OOP navigator to use C++/Java/C#

I think support for butterflies is what turns a text editor into a code editor (xkcd#378) XD
Post reply on HN