Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using gRPC for (local) inter-process communication (2021)

mpi-hd.mpg.de

Post reply on HN
Print view View original

71–80 of 99 posts

Re: Using gRPC for (local) inter-process communication (2021)

#71

Post by charleslmunger »

pjmlp wrote: post #12

> Using a full-featured RPC framework for IPC seems like overkill when the processes run on the same machine. That is exactly what COM/WinRT, XPC, Android Binder, D-BUS are. Naturally they have several optimisations for local execution.

You can run grpc over binder:

https://github.com/grpc/grpc-java/blob/master/binder/src/mai...

The overhead is low, and you get best practices like oneway calls and avoiding the transaction limit for free. It also comes with built in security policies for servers and clients.

Re: Using gRPC for (local) inter-process communication (2021)

#72

Post by imtringued »

bunderbunder wrote: post #55
jeffbee wrote: post #21

Earlier quoted context omitted.

Binder seriously underappreciated, IMHO. But I think it makes sense to use gRPC or something like it if there is any possibility that in the future an "IPC" will become an "RPC" to a foreign host. You don't want to be stuck trying to change an IPC into an RPC if it was foreseeable that it would eventually become remote due to scale.

This could possibly even be dead simple to accomplish if application-level semantics aren't being communicated by co-opting parts of the communication channel's spec. I think that this factor might be the ultimate source of my discomfort with standards like REST. Things like using HTTP verbs and status codes, and encoding parameters into the request's URL, mean that there's almost not even an option to choose a commu…

Nobody does REST, because nobody needs the whole hateoas shtick. When people say REST, they mean "HTTP API" and I'm not being pendantic here. The difference is very real because REST doesn't really have a reason to exist.

Re: Using gRPC for (local) inter-process communication (2021)

#73

Post by merb »

pjmlp wrote: post #45
merb wrote: post #44

Earlier quoted context omitted.

Btw. Modern windows also superports Unix domain sockets, so if you have an app that has another service that will run on the same machine or on a different one it is not so bad to use grpc over uds.

Nice idea, although it is still slower than COM. COM can run over the network (DCOM), inside the same computer on its own process (out-proc), inside the client (in-proc), designed for in-proc but running as out-proc (COM host). So for max performance, with the caveat of possibly damaging the host, in-proc will do it, and be faster than any kind of sockets.

Well if you need ipc to connect different languages, you will stay away from COM. Heck once you use anything but rust,c or c++ you should drop com. Even dotnet support for com is aweful. And if you ever written a outlook addin, than you will start hating com by yourself, thanks to god that Microsoft is going away from that

Re: Using gRPC for (local) inter-process communication (2021)

#74

Post by imtringued »

aseipp wrote: post #40
andy_ppp wrote: post #26

Earlier quoted context omitted.

JSON contains a description of the structure of the data that is readable by both machines and humans. JSON can certainly go wrong but it’s much simpler to see when it has because of this. GRPC is usually a binary black box that adds loads of developer time to upskill, debug, figure out error cases and introduces whole new classes of potential bugs. If you are building something that needs binary performance that GRP…

> JSON contains a description of the structure of the data that is readable by both machines and humans. No, it by definition does not, because JSON has no schema. Only your application contains and knows the (expected) structure of the data, but you literally cannot know what structure any random blob of JSON objects will have without a separate schema. When you read a random /docs page telling you "the structure of…

Here are some sad news for you: The flexibility of JSON and CBOR cannot be matched by any schema based system, because it is equivalent to giving up that advantage.

Sure, the removal of a field can cause an application level error, but that is probably the most benign form of failure there is. What's worse is when no error occurs and the data is simply reinterpreted to fit the schema. Then your database will slowly fill up with corrupted garbage data and you'll have to restore from a backup.

What you have essentially accomplished in your response is to miss the entire point.

There are also other problems with protobuf in the sense that the savings aren't actually as big as you'd expect. E.g. there is still costly parsing, the data transmitted over the wire isn't significantly smaller unless you have data that is a poor fit for JSON.

Re: Using gRPC for (local) inter-process communication (2021)

#75

Post by [deleted] »

imtringued wrote: post #74
aseipp wrote: post #40

Earlier quoted context omitted.

> JSON contains a description of the structure of the data that is readable by both machines and humans. No, it by definition does not, because JSON has no schema. Only your application contains and knows the (expected) structure of the data, but you literally cannot know what structure any random blob of JSON objects will have without a separate schema. When you read a random /docs page telling you "the structure of…

Here are some sad news for you: The flexibility of JSON and CBOR cannot be matched by any schema based system, because it is equivalent to giving up that advantage. Sure, the removal of a field can cause an application level error, but that is probably the most benign form of failure there is. What's worse is when no error occurs and the data is simply reinterpreted to fit the schema. Then your database will slowly f…

[deleted]

Re: Using gRPC for (local) inter-process communication (2021)

#76

Post by fwerner »

nicksnyder wrote: post #70
palata wrote: post #62

I have been in a similar situation, and gRPC feels heavy. It comes with quite a few dependencies (nothing compared to npm or cargo systems routinely bringing hundreds of course, but enough to be annoying when you have to cross-compile them). Also at first it sounds like you will benefit from all the languages that protobuf supports, but in practice it's not that perfect: some python package may rely on the C++ implem…

If you are looking for a lightweight Protobuf based RPC framework, check out https://connectrpc.com/ . It is part of the CNCF and is used and supported by multiple companies: https://buf.build/blog/connect-rpc-joins-cncf gRPC ships with its own networking stack, which is one reason why those libs are heavy. Connect libraries leverage each ecosystem's native networking stack (e.g. net/http in Go, NSURLSession in Swift…

[dead]

Re: Using gRPC for (local) inter-process communication (2021)

#77

Post by mdhb »

solarpunk wrote: post #41
DabbyDabberson wrote: post #29

Earlier quoted context omitted.

loved - in the past tense?

fuchsia was pretty deeply impacted by google layoffs iirc

I was just looking at the repo this week. It’s under heavy active development. It’s just intentionally not talked about much at the moment.

Re: Using gRPC for (local) inter-process communication (2021)

#78

Post by jsnell »

kentonv wrote: post #67
palata wrote: post #62

I have been in a similar situation, and gRPC feels heavy. It comes with quite a few dependencies (nothing compared to npm or cargo systems routinely bringing hundreds of course, but enough to be annoying when you have to cross-compile them). Also at first it sounds like you will benefit from all the languages that protobuf supports, but in practice it's not that perfect: some python package may rely on the C++ implem…

> it feels like it's essentially Cloudflare employees improving Cap'n Proto for Cloudflare. That's correct. At present, it is not anyone's objective to make Cap'n Proto appeal to a mass market. Instead, we maintain it for our specific use cases in Cloudflare. Hopefully it's useful to others too, but if you choose to use it, you should expect that if any changes are needed for your use case, you will have to make thos…

> What makes Google feel it's worthwhile to have 40ish(?) people working on an open source project that they don't actually use much themselves? Honest questions -- I don't know the answer, but I'd like to.

It's at least used for the public Google Cloud APIs. That by itself guarantees a rather large scale, whether they use gRPC in prod or not.

Re: Using gRPC for (local) inter-process communication (2021)

#79

Post by DanielHB »

At a project I worked at we were considering using protobuf for IPC between our desktop app and our network framework code which used different languages.

The performance was part of the reason (compared to serializing using JSON) but the main reason was just tooling support for automatic type checking. gRPC can generate types from a schema for all popular languages out there.

We ended up taking another route but I feel it is important to consider the existing tools ahead of any performance concerns for most cases

Re: Using gRPC for (local) inter-process communication (2021)

#80

Post by [deleted] »

pjmlp wrote: post #45
merb wrote: post #44

Earlier quoted context omitted.

Btw. Modern windows also superports Unix domain sockets, so if you have an app that has another service that will run on the same machine or on a different one it is not so bad to use grpc over uds.

Nice idea, although it is still slower than COM. COM can run over the network (DCOM), inside the same computer on its own process (out-proc), inside the client (in-proc), designed for in-proc but running as out-proc (COM host). So for max performance, with the caveat of possibly damaging the host, in-proc will do it, and be faster than any kind of sockets.

[deleted]
Post reply on HN