Live data from Hacker News

A new ProtoBuf generator for Go

vitess.io

61–70 of 78 posts

Re: A new ProtoBuf generator for Go

#61
post #5

I hadn't realized that Gogo was in such a bad spot with the upstream Go protobuf changes. There was lots of drama when the changes were made and I guess that overshadowed any optics I had on Gogo. Making vtprotobuf an additional protoc plugin seems like the Right Thing™, although it's a shame how complicated protoc commands end up becoming for mature projects. I'm pretty tempted to port Authzed over to this and run s…

Proto message unmarshal in Go for a small message should be 5 orders of magnitude below 20ms, shouldn't even begin to matter until you are sweating individual microseconds.

That's true if your program only does a single unmarshal at a time at a leisurely pace. And in a steady state situation, the memory trashing left behind each individual unmarshal call needs to be paid up by some poor future request.

I agree it's unlikely the difference here will be solely responsible for tipping the GP's request above 20ms, but the memory problems could reasonably ruin tail latencies.

Re: A new ProtoBuf generator for Go

#62
Funny timing, I've just written most of a TypeScript generator for protobufs. I learned about some fun corners of protobufs I didn't expect trying to pass the protouf conformance tests [1] (which this one passes, that's no mean feat!).

- If you write the same message multiple times, protobuf implementations should merge fields with a last write wins policy (repeated fields are concatenated). This includes messages in oneofs.

- For a boolean array, you're better off using a packed, repeated int64 (if wire size matters a lot). Protobuf bools use varint encoding meaning you need at least 2 bytes for every boolean, 1+ for the tag and type and 1 byte for the 0 or 1 value. With a repeated int64, you'd encode the tag and length in 2 varints, and then you get 64 bools per 8 bytes.

- Fun trivia: Varints take up a max of 10 bytes but could be implemented in 9 bytes. You get 7 bits per varint byte, so 9 bytes gets you 63 bits. Then you could use the most significant bit of the last byte to indicate if the last bit is 0 or 1. Learned by reading the Go varint implementation [2].

- Messages can be recursive. This is easy if you represent messages as pointers since you can use nil. It's a fair bit harder if you want to always use a value object for each nested message since you need to break cycles by marking fields as `T | undefined` to avoid blowing the stack. Figuring out the minimal number of fields to break cycles is an NP hard problem called the minimum feedback arc set[3].

- If you're writing a protobuf implementation, the conformance tests are a really nice way to check that you've done a good job. Be wary of implementations that don't implement the conformance tests.

[1]: https://github.com/protocolbuffers/protobuf/tree/master/conf...

[2]: https://github.com/golang/go/blob/master/src/encoding/binary...

[3]: https://en.wikipedia.org/wiki/Feedback_arc_set#Minimum_feedb...

Re: A new ProtoBuf generator for Go

#63
post #58

Earlier quoted context omitted.

I haven't looked in more detail, but one blocker is that `ProtoMethods() *methods` returns a private type, making it effectively unimplementable outside this package.

So, I thought this at one point, too. But it turns out that methods is a type alias to an unnamed type, so there's no package level privacy issues: https://github.com/protocolbuffers/protobuf-go/blob/v1.26.0/...

Oh huh, interesting, I've never seen that done before.

I'm struggling to understand what the rationale _for_ doing it is though. Maybe it's to avoid an import cycle?

Re: A new ProtoBuf generator for Go

#64
post #51

Earlier quoted context omitted.

It's not really suitable for latency-critical applications. EDIT: Fixed unfortunate typo

You can 100% write services with P999 P99 < 1ms, that's when you're going to want to switch it up.

Depending on workload, Go also does sub-1ms p99 pretty easily. I'm getting sub-1ms p99.9.

Re: A new ProtoBuf generator for Go

#65
post #56

Earlier quoted context omitted.

was the double-negative intentional? I've used Go for sub-millisecond needs. So 20ms seems like it would be a reasonable choice from where I'm sitting.

It was not intentional, thanks for asking...very unfortunate typo ;) Go doesn't give you control over inline vs indirect allocation, instead relying on escape analysis, which is notoriously finicky. Seemingly unrelated changes, along with compiler upgrades, can ruin your carefully optimized code. This is especially heinous because it uses a GC; unnecessary allocations have a disproportionately large impact on your ap…

But you think this impacts a 20ms budget? It’s mostly trivia to get sub 20ms p99 in Go.

Re: A new ProtoBuf generator for Go

#66
post #56

Earlier quoted context omitted.

was the double-negative intentional? I've used Go for sub-millisecond needs. So 20ms seems like it would be a reasonable choice from where I'm sitting.

It was not intentional, thanks for asking...very unfortunate typo ;) Go doesn't give you control over inline vs indirect allocation, instead relying on escape analysis, which is notoriously finicky. Seemingly unrelated changes, along with compiler upgrades, can ruin your carefully optimized code. This is especially heinous because it uses a GC; unnecessary allocations have a disproportionately large impact on your ap…

I don't know, I'm able to get 150k grpc q/sec with p99 sub 1ms. It's def better than G1 and CMS.

Re: A new ProtoBuf generator for Go

#67

Earlier quoted context omitted.

You can 100% write services with P999 P99 < 1ms, that's when you're going to want to switch it up.

Depending on workload, Go also does sub-1ms p99 pretty easily. I'm getting sub-1ms p99.9.

What are the proposed solutions to get better than that? C/Rust code? Assembly?

Re: A new ProtoBuf generator for Go

#68
post #58

Earlier quoted context omitted.

So, I thought this at one point, too. But it turns out that methods is a type alias to an unnamed type, so there's no package level privacy issues: https://github.com/protocolbuffers/protobuf-go/blob/v1.26.0/...

Oh huh, interesting, I've never seen that done before. I'm struggling to understand what the rationale _for_ doing it is though. Maybe it's to avoid an import cycle?

Yes, to avoid an import cycle or polluting the protoreflect API documentation with a rather large non-user-facing API surface.

Re: A new ProtoBuf generator for Go

#69
post #47
post #29

Earlier quoted context omitted.

Maybe, don't know. In what concerns me, although I like Rust, I only see it for scenarios where any kind of memory allocation is very precious, Ada/SPARK and MISRA-C style. I have been using GC languages with C++ like features, or polyglot codebases, for almost 20 years to think otherwise. Most of the time developers learn about new and miss out on the low level language features. It is a matter of balance, either tr…

Would you consider codecs or heavy numerical simulations to fall under those memory allocation scenarios that you'd use Rust for as well?

That is a good scenario, however you can still use languages like D, Nim, Swift, C#, F#, Go, among others for such scenarios.

For example, you can do codecs in C# on WinRT with .NET Native,

https://docs.microsoft.com/en-us/windows/uwp/audio-video-cam...

In the context of protobuf,

https://devblogs.microsoft.com/aspnet/grpc-performance-impro...

Back to Rust, yes it is a good option, I just wouldn't write the whole application on it, just specialized libraries.

Hence why I am looking forward to Rust/Windows efforts.

Re: A new ProtoBuf generator for Go

#70

Earlier quoted context omitted.

If your path is sensitive to 200us of latency you should probably optimize your application and tune your GC. Typically 200us for freeing all unreachable memory is not a big deal.

> If your path is sensitive to 200us of latency you should probably optimize your application and tune your GC. okay, you've done this, three years later and it's the same thing again since you need to accomodate the new features. your users haven't upgraded their computers. what do you do ?

Run a profiler and optimize again.
Post reply on HN