Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using gRPC for (local) inter-process communication (2021)

mpi-hd.mpg.de

Post reply on HN
Print view View original

21–30 of 99 posts

Re: Using gRPC for (local) inter-process communication (2021)

#21

Post by jeffbee »

pjmlp wrote: post #12

> Using a full-featured RPC framework for IPC seems like overkill when the processes run on the same machine. That is exactly what COM/WinRT, XPC, Android Binder, D-BUS are. Naturally they have several optimisations for local execution.

Binder seriously underappreciated, IMHO. But I think it makes sense to use gRPC or something like it if there is any possibility that in the future an "IPC" will become an "RPC" to a foreign host. You don't want to be stuck trying to change an IPC into an RPC if it was foreseeable that it would eventually become remote due to scale.

Re: Using gRPC for (local) inter-process communication (2021)

#23

Post by jeffbee »

andy_ppp wrote: post #10
pengaru wrote: post #8

There's a mountain of grpc-centric python code at $dayjob and it's been miserable to live with. Maybe it's less awful in c/c++, or at least confers some decent performance there. In python it's hot garbage.

Strongly agree, it’s has loads of problems, my least favourite being the schema is not checked in the way you might think, there’s not even a checksum to say this message and this version of the schema match. So when there’s old services/clients around and people haven’t versioned their schema’s safely (there was no mechanism for this apart from manually checking in PRs) you can get gibberish back for fields that sho…

There is an art to having forwards and backwards compatible RPC schemas. It is easy, but it is surprisingly difficult to get people to follow easy rules. The rules are as follows:

  1) Never change the type of a field
  2) Never change the semantic meaning of a field
  3) If you need a different type or semantics, add a new field
Pretty simple if you ask me.

Re: Using gRPC for (local) inter-process communication (2021)

#24

Post by jeffbee »

yetanotherdood wrote: post #22
jeffbee wrote: post #2

Interesting that it is taken on faith that unix sockets are faster than inet sockets.

Unix Domain Sockets are the standard mechanism for app->sidecar communication at Google (ex: Talking to the TI envelope for logging etc.)

Search around on Google Docs for my 2018 treatise/rant about how the TI Envelope was the least-efficient program anyone had ever deployed at Google.

Re: Using gRPC for (local) inter-process communication (2021)

#25

Post by andy_ppp »

jeffbee wrote: post #23
andy_ppp wrote: post #10

Earlier quoted context omitted.

Strongly agree, it’s has loads of problems, my least favourite being the schema is not checked in the way you might think, there’s not even a checksum to say this message and this version of the schema match. So when there’s old services/clients around and people haven’t versioned their schema’s safely (there was no mechanism for this apart from manually checking in PRs) you can get gibberish back for fields that sho…

There is an art to having forwards and backwards compatible RPC schemas. It is easy, but it is surprisingly difficult to get people to follow easy rules. The rules are as follows: 1) Never change the type of a field 2) Never change the semantic meaning of a field 3) If you need a different type or semantics, add a new field Pretty simple if you ask me.

If I got to choose my colleagues this would be fine, unfortunately I had people who couldn’t understand eventual consistency. One of the guys writing Go admitted he didn’t understand what a pointer was etc. etc.

Re: Using gRPC for (local) inter-process communication (2021)

#26

Post by andy_ppp »

jayd16 wrote: post #17
andy_ppp wrote: post #10

Earlier quoted context omitted.

Strongly agree, it’s has loads of problems, my least favourite being the schema is not checked in the way you might think, there’s not even a checksum to say this message and this version of the schema match. So when there’s old services/clients around and people haven’t versioned their schema’s safely (there was no mechanism for this apart from manually checking in PRs) you can get gibberish back for fields that sho…

You can trivially make breaking changes in a JSON blob too. GRPC has well documented ways to make non-breaking changes. If you're working somewhere where breaking schema changes go in with little fanfare and much debugging then I'm not sure JSON will save you. The only way to know is to dig through CLs? Write a test. There's also automated tooling to compare protobuff schemas for breaking changes.

JSON contains a description of the structure of the data that is readable by both machines and humans. JSON can certainly go wrong but it’s much simpler to see when it has because of this. GRPC is usually a binary black box that adds loads of developer time to upskill, debug, figure out error cases and introduces whole new classes of potential bugs.

If you are building something that needs binary performance that GRPC provides, go for it, but pretending there is no extra cost over doing the obvious thing is not true.

Re: Using gRPC for (local) inter-process communication (2021)

#27

Post by goalieca »

anilakar wrote: post #19

In addition to cloud connectivity, we've been using MQTT for IPC in our Linux IIoT gateways and touchscreen terminals and honestly it's been one of the better architectural decisions we've made. Implementing new components for specific customer use cases could not be easier and the component can be easily placed on the hardware or on cloud servers wherever it fits best. I don't see how gRPC could be any worse than th…

MQTT also lends itself to async very well. The event based approach is a real winner.

Re: Using gRPC for (local) inter-process communication (2021)

#28

Post by cherryteastain »

pengaru wrote: post #8

There's a mountain of grpc-centric python code at $dayjob and it's been miserable to live with. Maybe it's less awful in c/c++, or at least confers some decent performance there. In python it's hot garbage.

C++ generated code from protobuf/grpc is pretty awful in my experience.

Re: Using gRPC for (local) inter-process communication (2021)

#30

Post by eqvinox »

jeffbee wrote: post #24
yetanotherdood wrote: post #22

Earlier quoted context omitted.

Unix Domain Sockets are the standard mechanism for app->sidecar communication at Google (ex: Talking to the TI envelope for logging etc.)

Search around on Google Docs for my 2018 treatise/rant about how the TI Envelope was the least-efficient program anyone had ever deployed at Google.

Ok, now it sounds like you're blaming unix sockets for someone's shitty code...

No idea what "TI Envelope" is, and a Google search doesn't come up with usable results (oh the irony...) - if it's a logging/metric thing, those are hard to get to perform well regardless of socket type. We ended up using batching with mmap'd buffers for crash analysis. (I.e. the mmap part only comes in if the process terminates abnormally, so we can recover batched unwritten bits.)

Post reply on HN