Live data from Hacker News

Incremental Parsing in Go

dev-nonsense.com

21–30 of 78 posts

Re: Incremental Parsing in Go

#21

> a successful parse consumes at least one rune This avoids infinite loops?

Yes, there would be two outcomes for an attempted parsing. Either it succeeds and makes progress (and eventually terminates) or it fails and consumes nothing (and terminates because eventually you run out of parsers to try).

Re: Incremental Parsing in Go

#22

For max optimization, wouldn’t it be better to create a Rust or C library for parsing that Go links into? I personally don’t see the usefulness of trying to optimize Go itself too much as it’s handicapped by the runtime and garbage collection.

[deleted]

Re: Incremental Parsing in Go

#23
post #9

Earlier quoted context omitted.

This is a premature optimization, and keeping everything in the same language has benefits like greatly simplified tooling and building

It’s not a premature optimization - it’s deciding the maximum that the parser can be optimized in the future. Choosing Go sets a lower ceiling. > Keeping everything in the same language has benefits like greatly simplified tooling and building Surely there are other Go libraries that incorporate C, C++, or Rust? Also if both parsers existed and were equally easy to set up, and you were planning on doing a ton of pars…

Look at the current Makefile:

https://github.com/aretext/aretext/blob/main/Makefile

Build is literally a `go build ...` and install is `go install`. Adding any other language to the mix would make this a polyglot project and not be "equally easy to set up". The other question is, do both parsers exist? In this write-up they point to tree-sitter as a possibility which is a JS program that produces C code. This would be viable, but here's the author's take:

> I considered integrating tree-sitter, an incremental parsing library with parsers for many existing languages. However, running JavaScript to generate parsers and linking to a C library would have greatly complicated the build process. Today, aretext can be built on almost any platform using a single go install command. I’ve had users install aretext on ARM laptops, FreeBSD servers, Chromebooks, and Android phones. To maintain portability, I wanted a pure Go implementation.

So this wasn't some casual decision, but something they at least considered long enough to describe here.

And the parsing library itself is only around 1200 lines total (comments, blanks, and code). The parsers for each language add a lot more, of course, but should be roughly equivalent given the same library and interface. I imagine that if this project really takes off and performance becomes a real problem they can do the rewrite at that point. Right now, the code works, seems to work fast enough for its author and primary users, and it's trivial to install on any platform supported by Go. So yes, it would have been a premature optimization to complicate the build process, probably reduce the number of supported platforms (or greatly increase the effort to support the same number of platforms), just to have a slightly faster parser.

Re: Incremental Parsing in Go

#24
post #20
post #18

Earlier quoted context omitted.

One of Go's primary goals has always been compilation speed. Go started out in C, and was later (post-1.0) incrementally rewritten to be self-hosting. One of the authors (Ken Thompson) is also one of the co-creators of C. I would argue these guys know what they are doing.

I don’t know, not implementing generics when it was pretty obviously needed was a huge oversight, so I’m not sure. Also, the reason for compiler bootstrapping is more of a “beauty thing”, then practicality. It would definitely be faster in a low-level language, but I doubt it would matter as an end user.

You aren't sure if Ken Thompson knows what he's doing?

Re: Incremental Parsing in Go

#25

For max optimization, wouldn’t it be better to create a Rust or C library for parsing that Go links into? I personally don’t see the usefulness of trying to optimize Go itself too much as it’s handicapped by the runtime and garbage collection.

I've seen some real world example where Go was as fast or faster than Rust for CPU / io intensive task.

Go is a fast language even with a GC.

https://github.com/boyter/scc/#performance

Re: Incremental Parsing in Go

#26
post #20
post #18

Earlier quoted context omitted.

One of Go's primary goals has always been compilation speed. Go started out in C, and was later (post-1.0) incrementally rewritten to be self-hosting. One of the authors (Ken Thompson) is also one of the co-creators of C. I would argue these guys know what they are doing.

I don’t know, not implementing generics when it was pretty obviously needed was a huge oversight, so I’m not sure. Also, the reason for compiler bootstrapping is more of a “beauty thing”, then practicality. It would definitely be faster in a low-level language, but I doubt it would matter as an end user.

> not implementing generics when it was pretty obviously needed was a huge oversight

I get the desire for generics. I do a lot of C# and have used generics for a very many years. Yet I've been writing Go for around 6 or 7 years and other than in the beginning (when I was new to it) I haven't found myself missing them at all.

In other words, for many people the lack of generics comes across as an oversight. For others, including myself (again, a heavy generics user in C#) that really isn't the case. I write Go in the style of Go and it just hasn't been an issue.

Blanket statements are rarely true. YMMV.

Re: Incremental Parsing in Go

#27
post #16

Earlier quoted context omitted.

If you're making something requiring CPU optimization as a core feature, you might as well go with one of the fastest languages instead of handicapping your project from Day 1. Go is not considered one of the fastest. It's better for network or filesystem logic that is I/O limited.

This is kind of a test of how nuanced your understanding of programming languages can be. Rust with a bit of effort put into optimization will be faster than Go with a bit of effort put into optimization, it is true. However, you need to double-check your intuition for how big and how consequential the delta is, because I'd guesstimate it as roughly a factor of two, possibly a touch less. It is true that Rust does a…

Not sure if rust vs go would be the best example here. Rust vs Java would be a better one — go has a very primitive GC in comparison, and java does optimize hot loops to a higher degree, so a naive code base would be very hard to beat in a lower level language.

Re: Incremental Parsing in Go

#28

Earlier quoted context omitted.

C library for parsing? isn't it dangerous from security perspective?

It's just an example. The options are really Rust (what I'd prefer), C++, C, or perhaps something like Nim that compiles to C. If you’re trying to make an unoptimized parser, then use whatever you want.

[deleted]

Re: Incremental Parsing in Go

#29
post #20

Earlier quoted context omitted.

I don’t know, not implementing generics when it was pretty obviously needed was a huge oversight, so I’m not sure. Also, the reason for compiler bootstrapping is more of a “beauty thing”, then practicality. It would definitely be faster in a low-level language, but I doubt it would matter as an end user.

> not implementing generics when it was pretty obviously needed was a huge oversight I get the desire for generics. I do a lot of C# and have used generics for a very many years. Yet I've been writing Go for around 6 or 7 years and other than in the beginning (when I was new to it) I haven't found myself missing them at all. In other words, for many people the lack of generics comes across as an oversight. For others…

Well, it is not a blanket statement, it’s just the generic truth (pun not intended) based on decades of evolution of programming languages and a relatively expensive mistake for Java, which would have been a perfect opportunity to learn from.

Sure, it is seldom missed as an end user, but as a library user it is essential. That’s why map and the like had to be hard coded into the language, and why concurrent versions couldn’t be implemented for a long time in the language, the same way it was done for Java forever.

Post reply on HN