Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

gRPC: From service definition to wire format

kreya.app

Post reply on HN
Print view View original

11–20 of 33 posts

Re: gRPC: From service definition to wire format

#11

Post by jcgrillo »

jeffbee wrote: post #10
jcgrillo wrote: post #8

Earlier quoted context omitted.

I personally really like gRPC and protobufs. I think they strike a good balance between a number of indirectly competing objectives. However I completely agree with your observation that as soon as you move beyond a single source of truth for the .proto files it all goes to shit. I've seen some horrible things--generated code being committed to version control and copied between repos, .proto files duplicated and man…

I don't see why any of that would be necessary. There are simple rules for protobuf compatibility and people only need to follow them. Never re-use a field number to mean something else. Never change the type of a field. That's it. Those are the only rules. If you follow them you don't have to think about any of that stuff that you mentioned.

Absolutely! Forward and backward compatibility are one of the wonderful things about protobufs. And that all goes wrong when you try to define the interface in more than one place.

EDIT: also, although the wire protocol may tolerate unknown or missing data, almost always the application doesn't.

EDIT AGAIN: I'm not saying this is how it should be just that this is the low energy state the socio-technical system seems to arrive at over time. So ideally it should be simple but due to imperfect decisions it gets horribly complicated over time.

Re: gRPC: From service definition to wire format

#12

Post by cyberax »

Protobuf is good, but it's not perfect. The handling of "oneof" fields is weird and the Python bindings were written by drunk squirrels, enums are strange, etc.

gRPC is terrible, but ConnectRPC allows sane integration of PB with regular browser clients. Buf.build also has a lot of helpful tools, like backwards compatibility checking.

But it's not worse than other alternatives like Thrift. And waaaaaaaaaayyyyyy better than OpenAPI monstrosities.

Re: gRPC: From service definition to wire format

#13

Post by jeffbee »

jcgrillo wrote: post #11
jeffbee wrote: post #10

Earlier quoted context omitted.

I don't see why any of that would be necessary. There are simple rules for protobuf compatibility and people only need to follow them. Never re-use a field number to mean something else. Never change the type of a field. That's it. Those are the only rules. If you follow them you don't have to think about any of that stuff that you mentioned.

Absolutely! Forward and backward compatibility are one of the wonderful things about protobufs. And that all goes wrong when you try to define the interface in more than one place. EDIT: also, although the wire protocol may tolerate unknown or missing data, almost always the application doesn't. EDIT AGAIN: I'm not saying this is how it should be just that this is the low energy state the socio-technical system seems…

I fail to see how the application will even be aware of unknown data. Explain what practical problem could possibly arise if you think a message has 4 fields and I send you a fifth one.

Edited to reply to your edits: People who are just bozos with computers will never be kept from bozotry by any interchange format. If they lack any semblance of foresight then maybe they simply should get a different line of work. Postel's law is in force here. If you start sending me emails with extra headers my email program is never going to care. Protobufs are the same way.

Re: gRPC: From service definition to wire format

#14

Post by matja »

est wrote: post #7

> The contract-first philosophy gRPC/protobuf is largely a Google cult. I've seen too projects with complex business logic simply give up and embed JSON strings inside pb. Like WTF...? Everything was good in the begining, as long as everyone submits their .proto to a centralized repo. Once the one team starts to host their own, things get broken quickly. As it occured to me, gRPC could optionally just serve those .pr…

> As it occured to me, gRPC could optionally just serve those .proto files in the initial h2 handshake on the wire

Do you mean the reflection protocol, or some other .proto files?

Re: gRPC: From service definition to wire format

#15

Post by est »

jayd16 wrote: post #9
est wrote: post #7

> The contract-first philosophy gRPC/protobuf is largely a Google cult. I've seen too projects with complex business logic simply give up and embed JSON strings inside pb. Like WTF...? Everything was good in the begining, as long as everyone submits their .proto to a centralized repo. Once the one team starts to host their own, things get broken quickly. As it occured to me, gRPC could optionally just serve those .pr…

It does have discovery built in. Is that what you want?

you mean grpc.reflection.v1alpha.ServerReflection? Close enough, sadly not generally enabled.

Re: gRPC: From service definition to wire format

#16

Post by jcgrillo »

jeffbee wrote: post #13
jcgrillo wrote: post #11

Earlier quoted context omitted.

Absolutely! Forward and backward compatibility are one of the wonderful things about protobufs. And that all goes wrong when you try to define the interface in more than one place. EDIT: also, although the wire protocol may tolerate unknown or missing data, almost always the application doesn't. EDIT AGAIN: I'm not saying this is how it should be just that this is the low energy state the socio-technical system seems…

I fail to see how the application will even be aware of unknown data. Explain what practical problem could possibly arise if you think a message has 4 fields and I send you a fifth one. Edited to reply to your edits: People who are just bozos with computers will never be kept from bozotry by any interchange format. If they lack any semblance of foresight then maybe they simply should get a different line of work. Pos…

Apologies for the delay, this site appears to be rate limiting me. Yeah used correctly they're great. But they're almost never used correctly in practice. I agree this is bozotry in the extreme, but it's widespread. To avoid it all they'd need to do is read like 4 pages of well-written accessible documentation, but sadly that bar is too high. I don't blame protobufs! It's just that, somehow, what should be an elegant, simple system turns into a nightmare in practice. Every. Goddamn. Time. Not unlike when people try to use Kafka. That isn't to say the tool shouldn't be used, just that maybe we need a better way to organize/educate/hire engineers so they don't ruin things so badly. Or at least some way to impose an upper bound the damage they can do. Maybe there's some kind of regularization effect if you force everyone to work with Map JSON. Or maybe it's just the state everything devolves to eventually.

Re: gRPC: From service definition to wire format

#17

Post by pm90 »

My gripe with grpc is that it doesn’t play super well with kubernetes services… you have to take a little bit of care, you need to understand how k8s services work and you have to understand how load balancing in grpc works. Ideally I would want to use protobuf as an interchange format, and a “dumb” http server that understands that.

That being said… once you do configure it properly it can be a powerful tool. The complexity though is usually not worth it unless you’re at a certain scale.

Re: gRPC: From service definition to wire format

#18

Post by whs »

pm90 wrote: post #17

My gripe with grpc is that it doesn’t play super well with kubernetes services… you have to take a little bit of care, you need to understand how k8s services work and you have to understand how load balancing in grpc works. Ideally I would want to use protobuf as an interchange format, and a “dumb” http server that understands that. That being said… once you do configure it properly it can be a powerful tool. The co…

I wrote gRPC xDS server for Kubernetes that is configuration-free. Basically just load xDS client library into your code, then use xds:///servicename.namespace (unlike DNS, namespace is always required). It should be as lightweight and scales in similar way to the cluster DNS.

My company run this exact code in production since it was created in 2022. We probably have several times more than 1000 rps gRPC requests running internally including over the public internet for hybrid cloud connectivity. That being said, gRPC's xDS client is not always bugs-free.

https://github.com/wongnai/xds

Re: gRPC: From service definition to wire format

#19

Post by DaGardner »

pm90 wrote: post #17

My gripe with grpc is that it doesn’t play super well with kubernetes services… you have to take a little bit of care, you need to understand how k8s services work and you have to understand how load balancing in grpc works. Ideally I would want to use protobuf as an interchange format, and a “dumb” http server that understands that. That being said… once you do configure it properly it can be a powerful tool. The co…

Have you checked out https://connectrpc.com/ We are using this for the basic HTTP server/client compatibility. And that means we can also use from web a context without any proxy setup.

Re: gRPC: From service definition to wire format

#20

Post by selfmodruntime »

kjuulh wrote: post #3
tempest_ wrote: post #2

I like the idea of grpc because I wanted the contract but I tried it on a small service and I think I would avoid it in the future. Too many rough edges and features I didnt really need. I was using it in Rust and python mainly (maybe it is better in Go?) but it had a whole bunch of google stuff in there I didnt need. - Configuring the python client with a json string that did not seem to have a documented schema - E…

I think connect-rpc[0] strikes a good balance between normal http apis, and gRPC. It allows protobuf as json. So you could think of it as an opinionated http api spec. A health check would be just a call to an url /something.v1.MyService/MyMethod -d { "input": "something }. it works really well, and the tooling is pretty good, though it isn't that widely supported yet. Rust for one doesn't have an implementation. But…

+1 for connect-rpc. After go-micro‘s maintainer went off the rails, I ripped it all out and switched to connect.
Post reply on HN