A new ProtoBuf generator for Go • Hacker News Board

Re: A new ProtoBuf generator for Go

#21

Post by brandmeyer » Thu, Jun 03, 2021, 7:19 PM UTC

harikb wrote: post #9

brandmeyer wrote: post #8

Earlier quoted context omitted.

3% regression in QPS, 20% regression in CPU, and 5% regression in memory usage according to the article. Those are considerably worse than "5 orders of magnitude below".

GP meant 5 orders of magnitude below "20 ms". 20 ms is a lot of time. There is nothing one can do to a, say, a 1 kilo byte buffer that will cross 1 ms in any language. My own Go code doesn't cross more than few micros per message.

GP's root claim is that protobuf serialization/deserialization performance shouldn't matter, on an article where a user is specifically demonstrating that it does matter.

Re: A new ProtoBuf generator for Go

#22

Post by throwaway894345 » Thu, Jun 03, 2021, 7:19 PM UTC

morelisp wrote: post #6

jeffbee wrote: post #5

Earlier quoted context omitted.

Proto message unmarshal in Go for a small message should be 5 orders of magnitude below 20ms, shouldn't even begin to matter until you are sweating individual microseconds.

Until the GC kicks in and steals a full 200usec + a bunch of your throughput... (Holy shit, who is downvoting this? It's literally the whole article!)

If your path is sensitive to 200us of latency you should probably optimize your application and tune your GC. Typically 200us for freeing all unreachable memory is not a big deal.

Re: A new ProtoBuf generator for Go

#23

Post by geodel » Thu, Jun 03, 2021, 7:22 PM UTC

RhodesianHunter wrote: post #16

harikb wrote: post #11

Earlier quoted context omitted.

Properly written Go code (or even Java for that matter) will try to minimize allocations. For Java, unless I am mistaken pause-less GC is only offered by Azul - $$

>or even Java Just in case you may be unaware, the latest GCs for Java (Shenandoah, ZGC) are miles ahead of anything available for Go due to sheer age and manpower. Parallel and Pauseless are easily achievable in most cases.

> Latest GCs for Java (Shenandoah, ZGC) are miles ahead of anything available.

Beyond hyperbole, do you have any actual comparison of Go vs Java GC performance?

Re: A new ProtoBuf generator for Go

#24

Post by pjmlp » Thu, Jun 03, 2021, 7:23 PM UTC

p_l wrote: post #19

pjmlp wrote: post #14

Earlier quoted context omitted.

Not only that, there are other garbage collected languages like D, Nim and C# that offer the language features to do arenas without having to touch any C code. There is still so much education to do.

Aren't arenas old news in GC languages in general? Most of the time, their non-presence is due to general pools being just as good most of the time, or people simply not needing them that much with modern GC

Yes, so I really did not got how come such assertion was made.

Probably lack of experience with machine friendly code.

Re: A new ProtoBuf generator for Go

#25

Post by throwaway894345 » Thu, Jun 03, 2021, 7:24 PM UTC

pjmlp wrote: post #14

shoefindortz wrote: post #12

> Arenas are, however, unfeasible to implement in Go because it is a garbage collected language. If you are willing to use cgo, google already implemented one for gapid. https://github.com/google/gapid/tree/master/core/memory/aren...

Not only that, there are other garbage collected languages like D, Nim and C# that offer the language features to do arenas without having to touch any C code. There is still so much education to do.

Do I misunderstand what arenas are? I thought it was just "allocate this big array as a single allocation rather than N little allocations"? If so, how is that not supported in Go? (e.g., `arena := make([]Foo, 1000000000)`)

Re: A new ProtoBuf generator for Go

#26

Post by Zababa » Thu, Jun 03, 2021, 7:25 PM UTC

jen20 wrote: post #15

I'm not sure that the phrasing in the article is particularly fair: > The maintainers of Gogo, understandably, were not up to the gigantic task. I'm 99% sure they are "up to" (as in "capable of") doing so, they are just not "up for" it (as in, "will not do it").

They could be "not up to" because of lack of resources, probably time and/or money. I think that's what is implied, rather than lack of technical knowledge.

Re: A new ProtoBuf generator for Go

#27

Post by shoefindortz » Thu, Jun 03, 2021, 7:27 PM UTC

dimitrios1 wrote: post #20

pjmlp wrote: post #14

Earlier quoted context omitted.

Not only that, there are other garbage collected languages like D, Nim and C# that offer the language features to do arenas without having to touch any C code. There is still so much education to do.

I can't believe we've managed to have this lengthy of a discussion about GC languages and speed without anyone mentioning rust. Has HN turned a corner?

Rust has an arena allocator too[1], but it is implemented with 165(!!!) usages of unsafe. :)

[1] https://github.com/fitzgen/bumpalo

Re: A new ProtoBuf generator for Go

#28

Post by slimsag » Thu, Jun 03, 2021, 7:29 PM UTC

throwaway894345 wrote: post #25

pjmlp wrote: post #14

Earlier quoted context omitted.

Not only that, there are other garbage collected languages like D, Nim and C# that offer the language features to do arenas without having to touch any C code. There is still so much education to do.

Do I misunderstand what arenas are? I thought it was just "allocate this big array as a single allocation rather than N little allocations"? If so, how is that not supported in Go? (e.g., `arena := make([]Foo, 1000000000)`)

An arena allocator allows you to store many allocations _of different types_ in the same single chunk of memory, and then free all of them at one point in time.

Re: A new ProtoBuf generator for Go

#29

Post by pjmlp » Thu, Jun 03, 2021, 7:29 PM UTC

dimitrios1 wrote: post #20

pjmlp wrote: post #14

Earlier quoted context omitted.

Not only that, there are other garbage collected languages like D, Nim and C# that offer the language features to do arenas without having to touch any C code. There is still so much education to do.

I can't believe we've managed to have this lengthy of a discussion about GC languages and speed without anyone mentioning rust. Has HN turned a corner?

Maybe, don't know.

In what concerns me, although I like Rust, I only see it for scenarios where any kind of memory allocation is very precious, Ada/SPARK and MISRA-C style.

I have been using GC languages with C++ like features, or polyglot codebases, for almost 20 years to think otherwise.

Most of the time developers learn about new and miss out on the low level language features.

It is a matter of balance, either trying to do everything in a single language, or eventually write a couple of functions in a lower level language that are then used as building blocks for the rest of the application.

No need to throw away the ecosystem and developer tooling just to rewrite a data structure.

Re: A new ProtoBuf generator for Go

#30

Post by HNLogInsSuckAss » Thu, Jun 03, 2021, 7:29 PM UTC

hn_go_brrrrr wrote: post #3

PostThisTooFast wrote: post #2

Is there one for Kotlin yet? It's pretty pathetic that Google's own protocol lacks native support for its most popular operating system.

Yes: https://developers.google.com/protocol-buffers/docs/kotlintu...

Those must've been released only in the last couple of years. In 2019 there was still no Kotlin generator. The OP shouldn't have been modded down, because that is indeed pathetic.

https://medium.com/digitalfrontiers/a-dance-with-protocols-k...