Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

gRPC-Go Engineering Practices

grpc.io

Post reply on HN
Print view View original

71–80 of 85 posts

Re: gRPC-Go Engineering Practices

#71

Post by baby »

puzzle wrote: post #37
baby wrote: post #35

Earlier quoted context omitted.

it's a performance point of view. You will never beat a binary format to transfer data over the wire. Sure it's not as debugable, but if you want speed you don't have much choice. HTTP/2 did the same thing and ditched the textual format of HTTP/1.

Even with a binary format, you have a lot of options. gRPC has plans for binary logging, which Stubby has had for a decade or more. Google's Stubby clients and servers also have built-in HTTP handlers that let you do a lot of interactive debugging, like the /debug/requests interface in golang's net/trace package. Unexpectedly, I found that a much better troubleshooting experience than pouring through logs (also becau…

Not really. If you need to support multiple platforms and you need something secure (no DER or BER or...) then protobuf is pretty much unbeatable. Well-maintained Libraries in every language and good generated code based on a simple format.

Re: gRPC-Go Engineering Practices

#72

Post by ocdtrekkie »

Xorlev wrote: post #38
dmayle wrote: post #3

Earlier quoted context omitted.

Json is a serialization format. gRPC is both a serialization format and a DDL (data definition language). That means that you are storing your schema, which also happens to contain interoperability features. ...and the serialization format is more efficient.

Disclaimer: I work for Google, but not on gRPC. It's worth mentioning that gRPC is actually format-agnostic. While I can't say it does the best job of this (gRPC+protobuf works best), there's precedent for using any transport in gRPC. gRPC-Java includes examples of JSON serialization and Thrift serialization. gRPC is just the transport protocol (its self built on HTTP/2). That said, I'd highly recommend protobufs, th…

As a note, a disclaimer like this given without explanation could seem to indicate gRPC is a Google product even though apparently it no longer is. I actually had a conversation with the gRPC mailing list a few weeks back about how weirdly this has been communicated on their website as well. The linked blog post continues to refer to 'we' as 'Google'.

(For those curious, IIRC it is now been handed off to a subgroup of the Linux Foundation, called the Cloud Native Computing Foundation. The change does not appear to be well advertised on their website, which contains no reference to it.)

Re: gRPC-Go Engineering Practices

#73

Post by zimbatm »

rb808 wrote: post #2

Its a good time to ask, is gRPC any good? I'd love to standardize on stable middleware layer that handles multiple versions of clients and servers well. Rest with json really seems to work great for most things already. What is the advantage of gRPC - just more efficient?

Most of the comments here focus on Protobuf but the other half of gRPC is HTTP/2, and HTTP/2 is very complex.

Good luck implementing a half-decent HTTP/2 client or server library if your language doesn't support it already. This is easily a 6 months job.

The other issue with HTTP/2 is that most client libraries require TLS to work. Development environments become harder to setup. It becomes harder to sniff the traffic during debugging. In client/server scenarios where both are on the same machine this also creates unnecessary overhead.

Re: gRPC-Go Engineering Practices

#74

Post by weberc2 »

chickenbane wrote: post #57
virmundi wrote: post #42

Earlier quoted context omitted.

You're absolutely right. Auth0 shows that when compressed, protobufs don't provide a great benefit vs JSON. Something I've read from others is that gRPC is not the easiest thing to use directly in SPA. If you have services exposed to a web front end and mobile, double exposing a REST-ish API + gRPC might not be worth it. This is a problem with which I'm currently struggling. https://auth0.com/blog/beating-json-perfor…

Did you not finish the rest of the post? Yes, it's true that compressed json is not materially different in size vs protobufs. But when it comes to the speed of serializing and deserializing protobufs there is a huge gain across languages. So if you're using gRPC in backend environments you'll gain huge speed benefits.

That was already acknowledged earlier in the thread.

Re: gRPC-Go Engineering Practices

#75

Post by willvarfar »

There has been a flurry of gRPC posts on HN recently - it must be the new xml soap rest NoSQL fad!

NoSQL is an interesting parallel - Google publish map reduce and bigtable and amazon publish some influential papers and suddenly everyone is using NoSQL in order to be "web scale". Then it turns out that Google themselves were doing sql web scale and spanner and all that.

There's a risk that gRPC is the same? In chat yesterday ex-googlers said that Google was increasingly moving over to flatbuffer...

Personally I have an aversion for tools with generators. Harks back to the damage CORBA did me I guess... I also have a preference for plaintext eg JSON - so much easier to debug.

Oh well. Guess we're in the fashion business ... ;)

Re: gRPC-Go Engineering Practices

#76

Post by doh »

bruth wrote: post #70
doh wrote: post #45

Earlier quoted context omitted.

It has similar challenges than a monorepo. First challenge is, that you have to keep some kind of reference on what proto are you using within your project. What Google (and some others) do, is that they a) put all the proto files in a separate repo [0] and then generate them for each language separately (python [1], ...). This way you can use whichever proto file you need within your project, however you have to loa…

Thanks for the response! These are very good references. I like the breakdown of message and/or service definitions into separate files so it is easy to browse through the repository. I am assuming the version bump (v1 vs. v2) occurs when you introducing breaking changes to the API? I am aware that protobuf does a good job at allowing forwards and backwards compatibility at the message level..

Correct on the versioning. We internally treat protos as an immutable descriptions of our API interfaces. That means whenever we need to change anything (outside of bugs), be it adding a new field, or changing the order, renaming fields, changing types, ... we start a new version.

We also use a lot of inheritance of non-default types (timestamps, errors, ...) so it's important to make sure that we don't break anything for others.

Re: gRPC-Go Engineering Practices

#77

Post by doh »

marshallbrekka wrote: post #62
doh wrote: post #45

Earlier quoted context omitted.

It has similar challenges than a monorepo. First challenge is, that you have to keep some kind of reference on what proto are you using within your project. What Google (and some others) do, is that they a) put all the proto files in a separate repo [0] and then generate them for each language separately (python [1], ...). This way you can use whichever proto file you need within your project, however you have to loa…

The mono-repo of protos has been our approach, and has worked very well for us so far. Only difference is we publish the resulting code for all languages into a single artifact repo, which other projects use as a dependency.

Hw many languages do you use? Do you generate separate folders for each language or what does the structure looks like?

Re: gRPC-Go Engineering Practices

#78

Post by bruth »

doh wrote: post #76
bruth wrote: post #70

Earlier quoted context omitted.

Thanks for the response! These are very good references. I like the breakdown of message and/or service definitions into separate files so it is easy to browse through the repository. I am assuming the version bump (v1 vs. v2) occurs when you introducing breaking changes to the API? I am aware that protobuf does a good job at allowing forwards and backwards compatibility at the message level..

Correct on the versioning. We internally treat protos as an immutable descriptions of our API interfaces. That means whenever we need to change anything (outside of bugs), be it adding a new field, or changing the order, renaming fields, changing types, ... we start a new version. We also use a lot of inheritance of non-default types (timestamps, errors, ...) so it's important to make sure that we don't break anythin…

Interesting so adding new fields even.. I suppose there is quite a bit of planning and iterating on the interface before making it public to reduce churn? I understand doing that for backwards incompatible changes.

Re: gRPC-Go Engineering Practices

#79

Post by merb »

willvarfar wrote: post #9
rb808 wrote: post #2

Its a good time to ask, is gRPC any good? I'd love to standardize on stable middleware layer that handles multiple versions of clients and servers well. Rest with json really seems to work great for most things already. What is the advantage of gRPC - just more efficient?

Afraid I'm going to be contrary and old-fashioned and say I prefer JSON. Its never been problematic adding or extending JSON endpoints and its never been a problem using basic gzip compression on the fly either. And JSON endpoints are a damn sight easier to debug and wireshark and all the rest. I've spent a lot of time writing fast JSON serialization for various languages including Java etc; its staggering how ineffi…

> And JSON endpoints are a damn sight easier to debug and wireshark and all the rest.

https://github.com/google/protobuf/issues/3303

Re: gRPC-Go Engineering Practices

#80

Post by [deleted] »

weberc2 wrote: post #29
kajecounterhack wrote: post #4

Earlier quoted context omitted.

Versioning your API is one huge benefit and better done using proto/rpc. If you change your Json schema, have fun propagating that change to all clients without fear. Or hope you've built special infra to do that. Also "just more efficient" is a funny way to characterize the performance difference between just data bytes vs data + structure bytes (read: the gap is large). You gain in transmission and you gain during…

I fully agree, but want to point out that JSON can be compressed.

[deleted]
Post reply on HN