[dead]
Using gRPC for (local) inter-process communication (2021)
81–90 of 99 posts
Re: Using gRPC for (local) inter-process communication (2021)
#82Earlier quoted context omitted.
> JSON contains a description of the structure of the data that is readable by both machines and humans. No, it by definition does not, because JSON has no schema. Only your application contains and knows the (expected) structure of the data, but you literally cannot know what structure any random blob of JSON objects will have without a separate schema. When you read a random /docs page telling you "the structure of…
Here are some sad news for you: The flexibility of JSON and CBOR cannot be matched by any schema based system, because it is equivalent to giving up that advantage. Sure, the removal of a field can cause an application level error, but that is probably the most benign form of failure there is. What's worse is when no error occurs and the data is simply reinterpreted to fit the schema. Then your database will slowly f…
It's also worth noting CDDL [1], which adds schema-like utility to CBOR (and technically JSON.) We've started to use it in more places where we use CBOR.
Re: Using gRPC for (local) inter-process communication (2021)
#83Earlier quoted context omitted.
Nice idea, although it is still slower than COM. COM can run over the network (DCOM), inside the same computer on its own process (out-proc), inside the client (in-proc), designed for in-proc but running as out-proc (COM host). So for max performance, with the caveat of possibly damaging the host, in-proc will do it, and be faster than any kind of sockets.
Well if you need ipc to connect different languages, you will stay away from COM. Heck once you use anything but rust,c or c++ you should drop com. Even dotnet support for com is aweful. And if you ever written a outlook addin, than you will start hating com by yourself, thanks to god that Microsoft is going away from that
All new Windows API for over a decade are shipped as COM, iirc.
And COM ability to run in separate process is one of the things that makes it easier to run with different languages in my experience. Once you got over the hurdle of supporting COM, especially if you had support for type libraries and introspection-less COM, it was home free. Meanwhile I'm wary of dealing with a C++ library even if writing in C++
Re: Using gRPC for (local) inter-process communication (2021)
#84In addition to cloud connectivity, we've been using MQTT for IPC in our Linux IIoT gateways and touchscreen terminals and honestly it's been one of the better architectural decisions we've made. Implementing new components for specific customer use cases could not be easier and the component can be easily placed on the hardware or on cloud servers wherever it fits best. I don't see how gRPC could be any worse than th…
Trying to understand an existing IoT codebase is scarier than reading Lovecraft.
Out of curiosity, why were you using SSH reverse tunnel for IPC? Were you using virtualization inside your iot device and for some reason need a tunnel between the guests?
Re: Using gRPC for (local) inter-process communication (2021)
#85Earlier quoted context omitted.
This could possibly even be dead simple to accomplish if application-level semantics aren't being communicated by co-opting parts of the communication channel's spec. I think that this factor might be the ultimate source of my discomfort with standards like REST. Things like using HTTP verbs and status codes, and encoding parameters into the request's URL, mean that there's almost not even an option to choose a commu…
Nobody does REST, because nobody needs the whole hateoas shtick. When people say REST, they mean "HTTP API" and I'm not being pendantic here. The difference is very real because REST doesn't really have a reason to exist.
But most HTTP APIs that are commonly described as REST still use those parts of REST-over-JSON-over-HTTP, even if they aren't truly RESTful.
One thing I like about gRPC is that gRPC services use a single URL. And it doesn't ask you to define your own application error codes instead overloading HTTP status codes, thereby making it harder to distinguish application errors from communication channel errors. Things like that. But it still tightly couples itself to other parts of HTTP, particularly HTTP 3. That makes it impossible to use in a straightforward way in a number of scenarios, which then gives birth to a whole mess of reverse proxy shenanigans that people get into in an effort to avoid having to implement multiple copies of the same API.
Re: Using gRPC for (local) inter-process communication (2021)
#86Earlier quoted context omitted.
Nice idea, although it is still slower than COM. COM can run over the network (DCOM), inside the same computer on its own process (out-proc), inside the client (in-proc), designed for in-proc but running as out-proc (COM host). So for max performance, with the caveat of possibly damaging the host, in-proc will do it, and be faster than any kind of sockets.
Well if you need ipc to connect different languages, you will stay away from COM. Heck once you use anything but rust,c or c++ you should drop com. Even dotnet support for com is aweful. And if you ever written a outlook addin, than you will start hating com by yourself, thanks to god that Microsoft is going away from that
Good luck with that goal, after Longhorn went down, all its design ideas were resurfaced as COM, and since Vista, COM (alongside with its WinRT makeover), is the main API model in Windows.
Raw Win32 C API is stuck in Windows XP view of the world, with a few exceptions in some subsystems.
Yes COM tooling sucks, which is kind of sad given that it is the main API model in modern Windows, but I guess WinDev suffers from Stockhold syndrome.
AddIns are only moving away from COM, because otherwise they wouldn't work in your browser, so you get to use JavaScript and TypeScript instead, and on native Office they run inside a Webwidget, hurray for performance.
Re: Using gRPC for (local) inter-process communication (2021)
#87I have been in a similar situation, and gRPC feels heavy. It comes with quite a few dependencies (nothing compared to npm or cargo systems routinely bringing hundreds of course, but enough to be annoying when you have to cross-compile them). Also at first it sounds like you will benefit from all the languages that protobuf supports, but in practice it's not that perfect: some python package may rely on the C++ implem…
> it feels like it's essentially Cloudflare employees improving Cap'n Proto for Cloudflare. That's correct. At present, it is not anyone's objective to make Cap'n Proto appeal to a mass market. Instead, we maintain it for our specific use cases in Cloudflare. Hopefully it's useful to others too, but if you choose to use it, you should expect that if any changes are needed for your use case, you will have to make thos…
While you are correct the majority of Google services internally are Stubby-based, it isn't correct to say gRPC is not utilized inside Google. Cloud and external APIs are an obvious use case but also internal stuff that are also used in open source world use gRPC even when deployed internally. TensorFlow etc comes to mind...
So "not used much" at Google scale probably justifies 40 people.
Re: Using gRPC for (local) inter-process communication (2021)
#88This the typical situation where : “you can do it” doesn’t mean you should do it, and there is very little sense to advertise it, if at all.
Re: Using gRPC for (local) inter-process communication (2021)
#89> Using a full-featured RPC framework for IPC seems like overkill when the processes run on the same machine. That is exactly what COM/WinRT, XPC, Android Binder, D-BUS are. Naturally they have several optimisations for local execution.
ZeroMQ is kinda nice for this; just the right level of thin abstraction.
Re: Using gRPC for (local) inter-process communication (2021)
#90Earlier quoted context omitted.
> it feels like it's essentially Cloudflare employees improving Cap'n Proto for Cloudflare. That's correct. At present, it is not anyone's objective to make Cap'n Proto appeal to a mass market. Instead, we maintain it for our specific use cases in Cloudflare. Hopefully it's useful to others too, but if you choose to use it, you should expect that if any changes are needed for your use case, you will have to make thos…
While you are correct the majority of Google services internally are Stubby-based, it isn't correct to say gRPC is not utilized inside Google. Cloud and external APIs are an obvious use case but also internal stuff that are also used in open source world use gRPC even when deployed internally. TensorFlow etc comes to mind... So "not used much" at Google scale probably justifies 40 people.
> I think most people assume that gRPC is what Google is built on, therefore it must be good.
Dropbox[1], Netflix[2], Apple[3], Microsoft[4], Slack[5], and lots more are all either built on, or heavily use, and/or contribute to various pieces of gRPC and the ecosystem around it.
[1]: https://dropbox.tech/infrastructure/courier-dropbox-migratio...
[2]: https://netflixtechblog.com/practical-api-design-at-netflix-...
[3]: https://github.com/grpc/grpc-swift
[4]: https://learn.microsoft.com/en-us/aspnet/core/grpc/?view=asp...
[5]: https://slack.engineering/how-big-technical-changes-happen-a...