Varlink – IPC to replace D-Bus gradually in systemd

Post by pjmlp » Tue, Oct 01, 2024, 8:26 AM UTC

dathinab wrote: post #144

jchw wrote: post #5

Earlier quoted context omitted.

Unfortunately I concur. There are a lot of problems with D-Bus, but that it wasn't using JSON wasn't one of them. The tooling for D-Bus is terrible. It has composite types but they are very primitive, e.g. you can't name fields in a structure without extensions (there's like three different extensions for this.) A lot of the code generators can't actually handle real-world schemas. Now I understand that D-Bus itself…

Which makes me wonder how performance sensitive is the task D-Bus handles? Most task done over it seems to either fall into the "from time to time but not really performance sensitive at all" category or the "a mass of system events which can become huge" category. > miss out on the best benefits the best benefits of using binary serialization format I disagree (wrt. MsgPack), the biggest benefit is of binary formats…

Post by otabdeveloper4 » Tue, Oct 01, 2024, 12:08 PM UTC

gpderetta wrote: post #166

poettering wrote: post #157

Earlier quoted context omitted.

Not really. We use two text based formats for logging: BSD syslog, and systemd's structured logging (which is basically an env block, i.e. a key-value set, with some tweaks). Programs generate text logs, journald reads text logs hence. Programs that read from the journal get the text-based key/value stuff back, usually. (Yes, we then store the structure log data on disk in a binary format. Lookup indexes are just nas…

Sure the interface with the log might be text based, but my understanding is that the at rest format is binary and you need specialized tools to read it, standard unix grep is not going to cut it. Although I use strace all the time, I hardly ever look at the payload of read and write calls, although I could see why it would be useful. But given a binary protocol it wouldn't be terribly hard to build a tool that parse…

Post by cryptonector » Tue, Oct 01, 2024, 3:20 PM UTC

jchw wrote: post #237

cryptonector wrote: post #121

Earlier quoted context omitted.

BTW, Lustre's RPC system's serialization is very much based on C structs. It's receiver-makes-right to deal with endianness, for example. It's a pain, but it's also fast.

Making an RPC serialization system that is zero-overhead i.e. can use the same format on the wire as it does on disk is not a terrible idea. Capnp is the serialization format that I've been suggesting as a potential candidate and it is basically just taking the idea of C structs, dumping it into it's own schema language, and adding the bare minimum to get some protobuf-like semantics.

  - target 64-bit architectures
    (32-bit senders have to do work
     to encode, but 64-bit senders
     don't)
  - assume C-style struct packing
    rules in the host language
    (but not #pragma packed)
  - use an arena allocator
  - transmit {archflags, base pointer, data}
  - receiver makes right:
     - swab if necessary
     - fix interior pointers
     - fail if there are pointers
       to anything outside the
       received data
     - convert to 32-bit if the
       receiver is 32-bit

Post by rixed » Tue, Oct 01, 2024, 4:48 PM UTC

poettering wrote: post #125

newpavlov wrote: post #3

So Varlink requires a proper specification of message types, but then uses god damn JSON to serialize messages? Why would you do that? Apparently, we don't have enough wasted cycles in modern software, just add a bunch more in a fundamental IPC infrastructure.

The marshalling cost for JSON is negligible. Yes, it might be a bit slower than GVariant for example, but only by some fractional linear factor. And on small messages (which D-Bus currently always is, due to message size constraints enforced by broker) the difference is impossible to measure. To a point it really doesn't matter, in particular as JSON parsers have been ridiculously well optimized in this world. What d…

Post by accelbred » Tue, Oct 01, 2024, 5:26 PM UTC

poettering wrote: post #160

jeroenhd wrote: post #150

Earlier quoted context omitted.

Not a fan of JSON per se for this type of messaging, but I don't think binary parser+validator combinations are that much faster. Plus, this forces C(++) developers to actually validate the data they're exchanging rather than casting raw structs from data buffers, which solves a huge vulnerability. Then again, the lack of 64 bit integers makes the protocol design comically unsuited. I don't think the serialization/de…

The 64bit issue is certainly an issue, but very much overblown. First of all, in systemd, which is a heavy D-Bus user, we effectively IRL only send integers > 2^53 when we use UINT64_MAX as a special "niche" marker meaning "unlimited"/"unset"/"undefined". But the thing is that JSON has a proper concept for this: the "null" value (or simply not specifying a specific JSON field). Hence, in reasonably clean APIs this is…