Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

27.5KB language-agnostic WebGPU syntax highlighter

gpu-lexer.vercel.app

Post reply on HN
Print view View original

21–30 of 39 posts

Re: 27.5KB language-agnostic WebGPU syntax highlighter

#21

Post by silon42 »

avaer wrote: post #6
onion2k wrote: post #3

I don't think it's meant as something that's useful per se because highlighting 5m lines of code at once is unusual, and doing it in a non-deterministic way probably isn't especially helpful, but to show what a browser is capable of it's awesome.

The use case isn't highlighting a huge amount of code, but local re-entrant highlighting. You generally never need to highlight a whole file, only the part you're looking at. Which by definition must be fuzzy since you don't have the full source, the part you see will not parse as a complete program so you need to guess. This is actually one of the nastier parts of writing a highlighter, and it must be done for every…

No need to guess... For basic system highlighting, one can store the lexer context periodically (maybe per line) and invalidate it if edits are made before it. For more advanced stuff, you need a more complete parser anyway and do it asynchronously (LSP...).

Re: 27.5KB language-agnostic WebGPU syntax highlighter

#22

Post by vidarh »

silon42 wrote: post #21
avaer wrote: post #6

Earlier quoted context omitted.

The use case isn't highlighting a huge amount of code, but local re-entrant highlighting. You generally never need to highlight a whole file, only the part you're looking at. Which by definition must be fuzzy since you don't have the full source, the part you see will not parse as a complete program so you need to guess. This is actually one of the nastier parts of writing a highlighter, and it must be done for every…

No need to guess... For basic system highlighting, one can store the lexer context periodically (maybe per line) and invalidate it if edits are made before it. For more advanced stuff, you need a more complete parser anyway and do it asynchronously (LSP...).

That is a good optimization, but you still need to figure out the starting state at a given location, and for quite a lot of languages that is not solvable in the local case.

E.g. if you're looking at a page of Ruby, you can not in the general case know if it's inside or outside a quoted string, as the quote character can be any arbitrary character (and I really mean any. "% x " excluding the double-quotes is a valid Ruby quoted string where the quote character is space but you can e.g. pick a unicode codepoint you're not going to use anywhere else in the file)

So unless you parse from the start of the file, you're left with fuzzy matching, and that can be made good enough the vast majority of time.

Re: 27.5KB language-agnostic WebGPU syntax highlighter

#24

Post by vova_hn2 »

Years ago, long before the current "AI"/LLM-craze, I've heard someone saying that in the future programming will have less fixed, deterministic algorithms and more statistical/ML algorithms, because even in the case when writing a deterministic algorithm is possible, it is sometimes easier to collect training data and train a tiny model, then to write and maintain large codebase that does the same thing, but deterministically.

I wonder, if mass adoption of LLM code-generation will accelerate this process or not. On the one hand, it is now easier to "write" (LLM-generate) code then ever, so time/effort savings are not there anymore (probably, I'm not sure). On the other hand, now everyone uses LLMs so, I think, people are less deterred by non-determinism and statistical nature of ML models.

Let's imagine that you are making a CLI tool. What does a CLI tool do? Well, it accepts reads arguments, reads stdin, writes stdout and make syscalls. So, all possible inputs and outputs are very well defined. What if in the future it will be easier (and maybe even more natural) to ask an LLM to "imagine" tons of possible inputs and correct outputs for a tool that you are making and then train a tiny model, without writing or generating any code?

Re: 27.5KB language-agnostic WebGPU syntax highlighter

#25

Post by undefeated »

A genuinely impressive and novel use of machine learning? In this economy?

Looks really impressive, although one issue I do see is consistency. For example, it seems to highlight `null` as a value, unless it is on the left side of an `===` expression, since it probably hasn't seen that in training.

Was it ever actually tested on languages not in the training set? Would be interesting to see if a model this small can actually generalize...

Re: 27.5KB language-agnostic WebGPU syntax highlighter

#26

Post by vova_hn2 »

LatticeAnimal wrote: post #17

On linux (tried FF and chromium), the live demo renders all the text as black (no highlighting). On iOS (my phone), this works great. No errors in the console (maybe it fails silently?) Is this a linux issue / support issue?

I think, both FF and chromium on Linux disable WebGPU by default. Of course, it depends on where exactly did you get it (what kind of default settings people that packaged it for your distro used).

Just look up " enable webgpu" in you favorite search system or ask an LLM chatbot "How to enable WebGPU in ?", follow the instructions and it should work just fine.

Re: 27.5KB language-agnostic WebGPU syntax highlighter

#27

Post by weinzierl »

vova_hn2 wrote: post #24

Years ago, long before the current "AI"/LLM-craze, I've heard someone saying that in the future programming will have less fixed, deterministic algorithms and more statistical/ML algorithms, because even in the case when writing a deterministic algorithm is possible , it is sometimes easier to collect training data and train a tiny model, then to write and maintain large codebase that does the same thing, but determi…

"the future programming will have less fixed, deterministic algorithms and more statistical/ML algorithms"

I think the direction this will be going depends in part on how hardware prices will develop. If we continue with "hardware is cheap, don't think about it" like we did in the past decades I can see this happen. On the other hand, if we continue down the path we've recently taken, people who know when to choose which approach to their advantage will have a good and prosperous life.

Re: 27.5KB language-agnostic WebGPU syntax highlighter

#30

Post by lionkor »

undefeated wrote: post #25

A genuinely impressive and novel use of machine learning? In this economy? Looks really impressive, although one issue I do see is consistency. For example, it seems to highlight `null` as a value, unless it is on the left side of an `===` expression, since it probably hasn't seen that in training. Was it ever actually tested on languages not in the training set? Would be interesting to see if a model this small can…

How would it generalize? Let's say I use C, in which `class` is not a keyword, how would a model generalize such that it recognizes this?
Post reply on HN