Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incremental Parsing in Go

dev-nonsense.com

Post reply on HN
Print view View original

41–50 of 78 posts

Re: Incremental Parsing in Go

#41

Post by chrsig »

kaba0 wrote: post #39
richieartoul wrote: post #35

Earlier quoted context omitted.

I do a lot of “high throughput” stuff at work in both Go and Java, and the Go stuff is usually faster by default. Java tends to win for really naive programs where the author didn’t bother caring about performance or allocations at all, but if any care was put into it at all Go usually wins in my experience. The trope that Go’s GC is primitive in comparison to Javas is not really accurate. You can’t consider a langua…

Sure, Go can get away with more primitive GC exactly because it has “value types”, so less garbage is created. But they are still much worse, lower latency only means that they pause threads to get more breathing space if they have been allocating too heavily, they are absolutely not even close to the same league Java’s low latency ZGC does.

Do you have benchmarks you can share?

Re: Incremental Parsing in Go

#42

Post by skohan »

binwiederhier wrote: post #3

Interesting read. Thank you for sharing. I always found parsers fascinating and mystical. It seems like these parser functions (which i think are analogous to what Rob Pike calls state functions) are a common way to do parsing, though i know very little about it. I especially found the combinators intriguing, though I don't care much for the functional programming syntax in a language like Go. Anyway, thanks for shar…

Yeah parsers are fun! We did a recursive descent parser for a toy language in uni and I think it was one of the most illuminating and fun projects we did at school.

Lately I've been working on a tool to make it easy to implement a parser, and I end up using it for everything, because DSL's are so nice to work with.

Re: Incremental Parsing in Go

#43

Post by diffxx »

> The parsers produce a sequence to tokens, not a full syntax tree. Writing a tokenizer is much easier than parsing full syntax trees...Most other editors don’t construct the full syntax tree.

Syntax highlighting is nice, but fast semantic analysis is the real holy grail.

I have come to think that the best way to develop a new language would be to implement a text editor in that language before releasing the language. The editor should (ideally) have emacs and vim key bindings, though it would surely at least have the bindings that the language author uses. The compiler/interpreter for the language would embedded in the editor. This would allow for a much richer editing experience that goes beyond syntax highlighting. Indeed, the source code would become like a living document in the editor where they editor could display inline information about both syntax _and_ semantics. The editor need not be fancy. It could be written as a terminal application, kind of like a language specific nano or vim.

If the language/editor author is careful in how they design the editor, all of the syntactical and semantic tooling should be exportable into packages that can be consumed by other editors with plugin systems/lsp like vscode or neovim. Then the rich editing experience can be relatively easily exported to any other text editor. Tool authors would also then be able to write static analysis/linting/formatting/whatever tools on top of the semantic tooling that the language supports.

In some ways what I am describing is a more minimalist version of what one would get out of an image/IDE based language like smalltalk but the code representation would still remain as text files. The editor then becomes like a REPL except rather than being an ephemeral process, the REPL state is continually being written to disk and is resumable at any time from any computer capable of running the editor.

Re: Incremental Parsing in Go

#44

Post by kcartlidge »

chrsig wrote: post #40
kaba0 wrote: post #34

Earlier quoted context omitted.

Fair enough, I may not have used the correct word, but it is still a typical “told you” situation, both during development, after go’s initial appearance and ever since until it finally was decided that it should be indeed implemented.

There's a big misconception that the creators of go didn't want generics. They've stated a number of times that they didn't have a design that they all thought was adequate. After several years and attempts at a good enough proposal, Ian Lance Taylor put one out that was able to cross the finish line, and now we have generics.

> They've stated a number of times that they didn't have a design that they all thought was adequate

You know what, with all the 'discussions' in recent years about whether Go should have generics I'd actually lost track of that amongst all the noise. Which is irritating, as I do now remember the early conversations about it.

Thank you for the reminder.

Re: Incremental Parsing in Go

#45

Post by kaba0 »

chrsig wrote: post #41
kaba0 wrote: post #39

Earlier quoted context omitted.

Sure, Go can get away with more primitive GC exactly because it has “value types”, so less garbage is created. But they are still much worse, lower latency only means that they pause threads to get more breathing space if they have been allocating too heavily, they are absolutely not even close to the same league Java’s low latency ZGC does.

Do you have benchmarks you can share?

Well, cross-language benchmarking is always hard, but for purely testing the GC this is not too bad: https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

See how ahead Java is of any other managed language (and it doesn’t really make sense to do this benchmark with non-GCd languages)? Though this is done with the G1 GC, not the low-latency one - this default GC is more throughput oriented with a max pause time target goal. Also note how Java does use multiple times more memory, as it “postpones running GC when it knows it still will have enough time to collect all of it without running out of the target goal” - this is also the reason why java is quite ahead on “energy efficiency” reports as well. And also, GCs work better with more heap space.

Re: Incremental Parsing in Go

#46

Post by kaba0 »

fsdjkflsjfsoij wrote: post #37
kaba0 wrote: post #27

Earlier quoted context omitted.

Not sure if rust vs go would be the best example here. Rust vs Java would be a better one — go has a very primitive GC in comparison, and java does optimize hot loops to a higher degree, so a naive code base would be very hard to beat in a lower level language.

Java requires a much more advanced GC and JIT because Java programs tend to allocate a lot more and have extremely bad memory layout when you're not restricting yourself to primitives. Project Valhalla's value types will significantly improve the situation. Relying so heavily on the JIT also has other problems especially in programs that have widely varying execution paths.

Surely, that’s the incentive part for why the team spent many many work hours improving their GC - just because the JVM typically depends more on a good GC doesn’t make it any less useful - long running processes do make significant use of automatic memory management.

Also, Java’s GCs are moving/compacting GCs, so while the immediate memory representation is indeed inefficient, again, for long running processes Java will place often together-used objects physically close to each other, and will defragment the heap. But Valhalla can’t come soon enough, I agree.

> Relying so heavily on the JIT also has other problems especially in programs that have widely varying execution paths

Has it? I would think that an AOT program would have a worse time with widely varying execution paths, while a JIT compiler is free to reoptimise based on changing application state.

Re: Incremental Parsing in Go

#47

Post by maxbond »

diffxx wrote: post #43

> The parsers produce a sequence to tokens, not a full syntax tree. Writing a tokenizer is much easier than parsing full syntax trees...Most other editors don’t construct the full syntax tree. Syntax highlighting is nice, but fast semantic analysis is the real holy grail. I have come to think that the best way to develop a new language would be to implement a text editor in that language before releasing the language…

I think it would be sufficient & more valuable to implement a language server for your new language, which keeps you focused on the parts related to your language rather than the struggles of implementing an editor, and then you'll be able to drop into VSCode, neovim, etc.

Re: Incremental Parsing in Go

#48

Post by thechao »

maxbond wrote: post #47
diffxx wrote: post #43

> The parsers produce a sequence to tokens, not a full syntax tree. Writing a tokenizer is much easier than parsing full syntax trees...Most other editors don’t construct the full syntax tree. Syntax highlighting is nice, but fast semantic analysis is the real holy grail. I have come to think that the best way to develop a new language would be to implement a text editor in that language before releasing the language…

I think it would be sufficient & more valuable to implement a language server for your new language, which keeps you focused on the parts related to your language rather than the struggles of implementing an editor, and then you'll be able to drop into VSCode, neovim, etc.

Does anyone have an good intro tutorial for writing a language server in, say, C for some simple language? I find most of the docs a little too inscrutable to follow for just a bit of dabbling.

Re: Incremental Parsing in Go

#49

Post by diffxx »

maxbond wrote: post #47
diffxx wrote: post #43

> The parsers produce a sequence to tokens, not a full syntax tree. Writing a tokenizer is much easier than parsing full syntax trees...Most other editors don’t construct the full syntax tree. Syntax highlighting is nice, but fast semantic analysis is the real holy grail. I have come to think that the best way to develop a new language would be to implement a text editor in that language before releasing the language…

I think it would be sufficient & more valuable to implement a language server for your new language, which keeps you focused on the parts related to your language rather than the struggles of implementing an editor, and then you'll be able to drop into VSCode, neovim, etc.

If you write your own editor with language server features, writing a language server implementation _should_ (dangerous word) be relatively easy. By writing the editor in your language, you prove that your language is actually useful for a real world problem. You also have the flexibility to add any functionality that you wish without contorting your thinking to whatever may be imposed on you by the lsp model.

It isn't necessarily clear to me that writing a language server implementation is that much easier than writing an editor. To be fair, I have personally done neither (though I have written REPLs) so perhaps I am wildly off in my estimation.

Re: Incremental Parsing in Go

#50

Post by maxbond »

diffxx wrote: post #49
maxbond wrote: post #47

Earlier quoted context omitted.

I think it would be sufficient & more valuable to implement a language server for your new language, which keeps you focused on the parts related to your language rather than the struggles of implementing an editor, and then you'll be able to drop into VSCode, neovim, etc.

If you write your own editor with language server features, writing a language server implementation _should_ (dangerous word) be relatively easy. By writing the editor in your language, you prove that your language is actually useful for a real world problem. You also have the flexibility to add any functionality that you wish without contorting your thinking to whatever may be imposed on you by the lsp model. It is…

I don't really disagree, but would point out that, for the languages I hack on, they're DSLs that wouldn't be appropriate to write an editor in. I agree that you should be writing code in your language & solving problems with it though, even before there's a working implementation. It does really help hone your vision and focus on the problems you're solving.

I've not written a language server either, and I'm positive it's one or two orders of magnitude more difficult than a simple editor, but I think the time is better spent for many projects because it gets you into the nitty gritty mechanics of your language & sets you up for a good developer experience. If your language specializes in writing things like editors though, that would certainly not be the case.

Tangentially, my language development hot take is that YAML is a really great platform to start with, so you can focus on prototyping your runtime & semantics and kick the can on parsing & syntax. Parsing takes up a lot of time and you might not know if this language idea is worthwhile or not yet. Additionally, YAML is pretty darn good and you may never need to move off of it.

Post reply on HN