Using gRPC for (local) inter-process communication (2021)
1–10 of 99 posts
Re: Using gRPC for (local) inter-process communication (2021)
#2Interesting that it is taken on faith that unix sockets are faster than inet sockets.
Re: Using gRPC for (local) inter-process communication (2021)
#3> Using a full-featured RPC framework for IPC seems like overkill when the processes run on the same machine. However, if your project anyway exposes RPCs for public APIs or would benefit from a schema-based serialisation layer it makes sense to use only one tool that combines these—also for IPC.
It might make sense. Usually, if you're using IPC, you need it to be as fast as possible and there are several solutions that are much faster.
Re: Using gRPC for (local) inter-process communication (2021)
#4Interesting that it is taken on faith that unix sockets are faster than inet sockets.
Are there resources suggesting otherwise?
Re: Using gRPC for (local) inter-process communication (2021)
#5Interesting that it is taken on faith that unix sockets are faster than inet sockets.
Tell me more, I know nothing about IPC
Re: Using gRPC for (local) inter-process communication (2021)
#6Interesting that it is taken on faith that unix sockets are faster than inet sockets.
That's because it's logical that implementing network capable segmentation and flow control is more costly than just moving data with internal, native structures. And looking up random benchmarks yields anything from equal performance to 10x faster for Unix domain.
Re: Using gRPC for (local) inter-process communication (2021)
#7> Using a full-featured RPC framework for IPC seems like overkill when the processes run on the same machine. However, if your project anyway exposes RPCs for public APIs or would benefit from a schema-based serialisation layer it makes sense to use only one tool that combines these—also for IPC. It might make sense. Usually , if you're using IPC, you need it to be as fast as possible and there are several solutions…
I tend to agree. Usually you want as fast as possible. Sometimes you don't though.
E.g. Kythe (kythe.io) was designed so that its individual language indexers run with a main driver binary written in Go, and then a subprocess binary written in.... whatever. There's a requirement to talk between the two, but it's not really a lot of traffic (relative to e.g. the CPU cost of the subprocess doing compilation).
So what happens in practice is that we used Stubby (like gRPC, except not public), because it was low overhead* to write the handler code for it on both ends, and got us some free other bits as well.
* Except when it wasn't lol. It worked great for the first N languages written in langs with good stubby support. But then weird shit (for Google) crawled out of the weeds that didn't have stubby support, so there's some handwaving going on for the long tail.
Re: Using gRPC for (local) inter-process communication (2021)
#8There's a mountain of grpc-centric python code at $dayjob and it's been miserable to live with. Maybe it's less awful in c/c++, or at least confers some decent performance there. In python it's hot garbage.
Re: Using gRPC for (local) inter-process communication (2021)
#9> Using a full-featured RPC framework for IPC seems like overkill when the processes run on the same machine. However, if your project anyway exposes RPCs for public APIs or would benefit from a schema-based serialisation layer it makes sense to use only one tool that combines these—also for IPC. It might make sense. Usually , if you're using IPC, you need it to be as fast as possible and there are several solutions…
I'm not even sure that I'd say usually. Most of the time you're just saying "hey daemon, do this thing that you're already preconfigured for".
Re: Using gRPC for (local) inter-process communication (2021)
#10There's a mountain of grpc-centric python code at $dayjob and it's been miserable to live with. Maybe it's less awful in c/c++, or at least confers some decent performance there. In python it's hot garbage.
Strongly agree, it’s has loads of problems, my least favourite being the schema is not checked in the way you might think, there’s not even a checksum to say this message and this version of the schema match. So when there’s old services/clients around and people haven’t versioned their schema’s safely (there was no mechanism for this apart from manually checking in PRs) you can get gibberish back for fields that should contain data. It’s basically just a binary blob with whatever schema the client has overlaid so debugging is an absolute pain. Unless you are Google scale use a text based format like JSON and save yourself a lot of hassle.