181–190 of 273 posts
But what is the excuse to start from scratch? Rather than a compat layer that convinces the world that raw your interfaces, libraries and clients are so much nicer to work with that the other thing should be demoted to compat layer eventually?
Earlier quoted context omitted.
> lingua-franca serialization format it isn't, fundamentally can't be as it has no support for binary blobs and base64 encoded strings are a pretty terrible solution for this and the space overhead of many small number can also be pretty bad and while a lot of IPC use cases are so little performance sensitive that JSON is more then good enough and large blobs are normally not send over messages (instead you e.g. send…
The protocol supports “upgrading” requests. If your service relies on sending large binary blobs (over this? Why?), then it doesn’t have to be done with JSON. For example, the metadata of the blob could be returned via JSON, then the request is “upgraded” to a pure binary pipe and the results read as-is.
binary blobs is just the biggest example and was only mentioned in relation to the "lingua franca" argument, many other common things are also larger in JSON. Only if you have many larger not escaped utf-8 strings does this overheads amortize. E.g. uuids are something not uncommonly send around and it's 17 bytes in msgpack as a bin value and 38 bytes in json (not inlcuding `:` and `,` ). That 211% the storage cost. Multiply it with something going on and producing endless amounts of events (e.g. some unmount/mount loop) and that difference can matter.
Through yes for most use cases this will never matter.
Earlier quoted context omitted.
One can view the binary log files using journalctl. Per https://systemd.io/JOURNAL_FILE_FORMAT/ , the benefits of the binary format are: The systemd journal stores log data in a binary format with several features: Fully indexed by all fields Can store binary data, up to 2^64-1 in size Seekable Primarily append-based, hence robust to corruption Support for in-line compression Support for in-line Forward Secure Sealin…
> Primarily append-based, hence robust to corruption It's so robust, it doesn't even let you modify the journal if you want to (e.g. https://github.com/systemd/systemd/issues/20673 ). > Support for in-line compression Mind that journald only supports compressing single lines, but not the whole journal ( https://github.com/systemd/systemd/issues/31358 ), which is pretty limiting.
In what situations is it a harder problem than this?
Earlier quoted context omitted.
Data format is absolutely important for operating system IPC. JSON is raw text. Messages occupy more RAM than a binary format, which increases your working set, and it requires at least some nontrivial parsing. This ultimately limits the number of messages you can send per second.
It's not that simple: [1.2,1.3,1.4] is 13 bytes, the equivalent binary storage is 3*sizeof(double) (24 bytes) + whatever overhead exists for representing a dynamic array in your language
OTOH this is pretty cherry picked. It is questionable if you really need floating point numbers if your numbers are all this small and low precision, but in actuality unless this comprises your entire message, you'll probably still lose to JSON overhead eventually anyways, whereas non-self-describing serialization like capnp has effectively no overhead by default. The advantage of encoding as doubles is that it is predictable and won't explode into taking massively more, its always the same size per number. If you want it to be smaller for some reason, compression is always an option, though I suspect for IPC it's the wrong thing to be concerned about.
So Varlink requires a proper specification of message types, but then uses god damn JSON to serialize messages? Why would you do that? Apparently, we don't have enough wasted cycles in modern software, just add a bunch more in a fundamental IPC infrastructure.
It's a small format, it has limited pitfalls, it's fast to marshal and unmarshal (and with simd gets even more fast), it's deflate-friendly, would performance be an issue they can pack that through CBOR or msgpack or what-have-you.
Earlier quoted context omitted.
The protocol supports “upgrading” requests. If your service relies on sending large binary blobs (over this? Why?), then it doesn’t have to be done with JSON. For example, the metadata of the blob could be returned via JSON, then the request is “upgraded” to a pure binary pipe and the results read as-is.
not one large blob, many small events binary blobs is just the biggest example and was only mentioned in relation to the "lingua franca" argument, many other common things are also larger in JSON. Only if you have many larger not escaped utf-8 strings does this overheads amortize. E.g. uuids are something not uncommonly send around and it's 17 bytes in msgpack as a bin value and 38 bytes in json (not inlcuding `:` an…
Yes, there’s a more optimal format than hex-encoding UUID values. However it simply does not matter for any use case this targets.
16 bytes vs 38 bytes is completely meaningless in the context of a local process sending a request to a local daemon. It’s meaningless when making a HTTP request as well, unfortunately.
I’d have loved Arrow to be the format chosen, but that’s not lowering the barrier to entry much.
Earlier quoted context omitted.
It's the lingua-franca serialization format that has bindings to every programming language already, doesn't require agreeing on the payload layout beforehand, is beyond overkill for the task being asked of it, and is human debuggable over the wire. I'm begging you to read the format it's replacing. https://dbus.freedesktop.org/doc/dbus-specification.html Ctrl+f Type System. JSON is beautiful, elegant, and minimal by…
JSON isn't simple. It's simplistic. It's too simple. By refusing to acknowledge the complexity of the real world at the protocol layer, varlink shifts the burden of dealing with the complexity from one transport layer to N ad hoc application layers, making us all worse off.
Christ almighty, I have seldom seen a technical proposal as bad as this one. I want off Mr. Poettering's wild ride. No 64 bit integers? No file descriptor transport? No message sequence numbers? Just cut it out already. Dbus doesn't do something? Extend dbus. Don't make some other random and worse things.
Are these required or useful in this context?
>Dbus doesn't do something? Extend dbus.
How are you so certain that this is the best use of resources, do you have personal XP in the problem domain?
> Don't make some other random and worse things.
How did you determine that these changes are "worse" or "random?"
Earlier quoted context omitted.
The marshalling cost for JSON is negligible. Yes, it might be a bit slower than GVariant for example, but only by some fractional linear factor. And on small messages (which D-Bus currently always is, due to message size constraints enforced by broker) the difference is impossible to measure. To a point it really doesn't matter, in particular as JSON parsers have been ridiculously well optimized in this world. What d…
The retrospective on the Xi editor project has a few notes on issues with JSON (at least in the context of Rust and Swift). This is from someone who initially seemed to have a very similar perspective to you. See the JSON section of the post: https://raphlinus.github.io/xi/2020/06/27/xi-retrospective.h...