Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Varlink – IPC to replace D-Bus gradually in systemd

varlink.org

Post reply on HN
Print view View original

141–150 of 273 posts

Re: Varlink – IPC to replace D-Bus gradually in systemd

#141

Post by Dunedan »

solarengineer wrote: post #134
markasoftware wrote: post #103

Earlier quoted context omitted.

> Lennart pointed out the fact you can see readable messages via strace to be a benefit of json. from the guy who brought you binary logfiles!

One can view the binary log files using journalctl. Per https://systemd.io/JOURNAL_FILE_FORMAT/ , the benefits of the binary format are: The systemd journal stores log data in a binary format with several features: Fully indexed by all fields Can store binary data, up to 2^64-1 in size Seekable Primarily append-based, hence robust to corruption Support for in-line compression Support for in-line Forward Secure Sealin…

> Primarily append-based, hence robust to corruption

It's so robust, it doesn't even let you modify the journal if you want to (e.g. https://github.com/systemd/systemd/issues/20673).

> Support for in-line compression

Mind that journald only supports compressing single lines, but not the whole journal (https://github.com/systemd/systemd/issues/31358), which is pretty limiting.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#143

Post by throwaway984393 »

cryptonector wrote: post #115
NekkoDroid wrote: post #27

Earlier quoted context omitted.

systemd has been using varlink for a while now along side dbus, its not something just now being introduced. They don't have a problem living side-by-side.

It's more code, more bloat, more complexity, and more attack surface area.

[dead]

Re: Varlink – IPC to replace D-Bus gradually in systemd

#144

Post by dathinab »

jchw wrote: post #5
newpavlov wrote: post #3

So Varlink requires a proper specification of message types, but then uses god damn JSON to serialize messages? Why would you do that? Apparently, we don't have enough wasted cycles in modern software, just add a bunch more in a fundamental IPC infrastructure.

Unfortunately I concur. There are a lot of problems with D-Bus, but that it wasn't using JSON wasn't one of them. The tooling for D-Bus is terrible. It has composite types but they are very primitive, e.g. you can't name fields in a structure without extensions (there's like three different extensions for this.) A lot of the code generators can't actually handle real-world schemas. Now I understand that D-Bus itself…

Which makes me wonder how performance sensitive is the task D-Bus handles?

Most task done over it seems to either fall into the "from time to time but not really performance sensitive at all" category or the "a mass of system events which can become huge" category.

> miss out on the best benefits the best benefits of using binary serialization format

I disagree (wrt. MsgPack), the biggest benefit is of binary formats is that you can ship binary blobs without any encoding (mainly space) overhead while for JSON this can easily be ~33%+ space overhead. And even for just e.g. numbers the space overhead can be bad. I'm not sure what you see as the biggest benefit but not having a minimalist structure isn't a property specific to binary formats (through more common there, and MsgPack doesn't require field names to be included). And bit fiddling micro space optimizations aren't that grate/as much a benefit as drawback. And as a system bus handles (emits) potentially a _ton_ of events saving 33% can matter I think.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#145

Post by trissylegs »

akira2501 wrote: post #60
arianvanp wrote: post #25

Earlier quoted context omitted.

CBOR is the obvious choice here. I've suggested it before and the systemd folks didn't seem completely opposed to it. Also because cbor parser is already in the dependencies due to FIDO2 disk unlocking

I genuinely dislike CBOR. Formats which require me to calculate the length of the message before sending it lead to bad library design which often leads to easily compromised code. Add in an "indefinite length" option and you've got potentially unbounded client memory usage to watch out for. As if that wasn't enough you get extensible tags so the meaning of any message is entirely dependent on the context it was sent…

CBOR does have indefinite length arrays, maps, byte strings and text strings : https://www.rfc-editor.org/rfc/rfc8949.html#name-indefinite-...

Re: Varlink – IPC to replace D-Bus gradually in systemd

#146

Post by skavi »

poettering wrote: post #125
newpavlov wrote: post #3

So Varlink requires a proper specification of message types, but then uses god damn JSON to serialize messages? Why would you do that? Apparently, we don't have enough wasted cycles in modern software, just add a bunch more in a fundamental IPC infrastructure.

The marshalling cost for JSON is negligible. Yes, it might be a bit slower than GVariant for example, but only by some fractional linear factor. And on small messages (which D-Bus currently always is, due to message size constraints enforced by broker) the difference is impossible to measure. To a point it really doesn't matter, in particular as JSON parsers have been ridiculously well optimized in this world. What d…

The retrospective on the Xi editor project has a few notes on issues with JSON (at least in the context of Rust and Swift).

This is from someone who initially seemed to have a very similar perspective to you.

See the JSON section of the post: https://raphlinus.github.io/xi/2020/06/27/xi-retrospective.h...

Re: Varlink – IPC to replace D-Bus gradually in systemd

#147

Post by dathinab »

Spivak wrote: post #111
userbinator wrote: post #100

Earlier quoted context omitted.

It doesn't matter what the rate is. Of all the things they could do, they decided to choose one of the worst possible formats for IPC data[1], maybe to pander to the web crowd. [1] Maybe XML would be a close competitor.

It's the lingua-franca serialization format that has bindings to every programming language already, doesn't require agreeing on the payload layout beforehand, is beyond overkill for the task being asked of it, and is human debuggable over the wire. I'm begging you to read the format it's replacing. https://dbus.freedesktop.org/doc/dbus-specification.html Ctrl+f Type System. JSON is beautiful, elegant, and minimal by…

> lingua-franca serialization format

it isn't, fundamentally can't be as it has no support for binary blobs

and base64 encoded strings are a pretty terrible solution for this

and the space overhead of many small number can also be pretty bad

and while a lot of IPC use cases are so little performance sensitive that JSON is more then good enough and large blobs are normally not send over messages (instead you e.g. send a fd) there are still some use-cases where you use IPC to listen to system events and you have a situation where a pretty insane amount of them is emitted, in which case JSON might come back and bite you (less due to marshaling performance and more due to the space overhead which depending on the kind of events can easily be 33%+).

But what I do not know is how relevant such edge cases are.

Probably the biggest issue might be IPC system mainly used on 64bit systems not being able to properly represent all 64 bit integers .... but probably that is fine too.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#148

Post by dathinab »

jsnell wrote: post #55
ptx wrote: post #53

Earlier quoted context omitted.

> Performance looks like a total non-issue for this use case. The presentation brings up performance as one of the problems with the D-Bus though, so it seems like it is an issue.

Fair. What I meant is that the performance delta between JSON and any other interchange data format should be a non-issue in this context. I would be quite surprised to hear that any performance problems systemd has with D-Bus was with serialization/de-serialization.

> performance delta between JSON and any other interchange data format should be a non-issue

it's hard to say, for most D-Bus taks perf. is an absolute non issues, but still you have bus-1 and all kind of tries to make D-Bus perform faster, reason is there are edge cases where it matters

and even if the marshaling speed might not matter the size can, like if something goes really wrong and tens of thousand of system events get spammed the difference between the memory usage of in-flight messages of a compact format vs. json can make the difference between memory pressure effectively killing your server and you recovering reliable

though what is an open question is how relevant that is in pracive

and that is quite hard to say without inside information for companies running systems where such a thing could happen on a scale where such a thing does happen (e.g. google)

Re: Varlink – IPC to replace D-Bus gradually in systemd

#149

Post by RandomThoughts3 »

cryptonector wrote: post #113
RandomThoughts3 wrote: post #84

Earlier quoted context omitted.

> The perceived complexity of ASN.1 comes from the schema language and specifications written in the convoluted telco/ITU-T style (and well, the 80's type system that has ~8 times two different types for “human readable string”). I can’t resist pointing that it’s basically a longer way of saying quite complicated and not very efficient.

> I can’t resist pointing that it’s basically a longer way of saying quite complicated and not very efficient. That's very wrong. ASN.1 is complicated because it's quite complete by comparison to other syntaxes, but it's absolutely not inefficient unless you mean BER/DER/CER, but those are just _some_ of the encoding rules available for use with ASN.1. To give just one example of "complicated", ASN.1 lets you specify…

> That's very wrong. ASN.1 is complicated because it's quite complete by comparison to other syntaxes

So, it's quite complicated. Yes, what I have been saying from the start. If you start the conversation by "you can define a small subset of this terrible piece of technology which is bearable", it's going to be hard convincing people it's a good idea.

> Cavalier attitudes like "ASN.1 is too complicated" lead to bad results.

I merely say quite complicated not too complicated.

Still, ASN.1 is a telco protocol through and through. It shows everywhere: syntax, tooling. Honestly, I don't see any point in using it unless it's required by law or by contract (I had to, I will never again).

> but it's absolutely not inefficient unless you mean BER/DER/CER, but those are just _some_ of the encoding rules available for use with ASN.1.

Sorry, I'm glade to learn you can make ASN.1 efficient if you are a specialist and now what you are doing with the myriad available encodings. It's only inefficient in the way everyone use it.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#150

Post by jeroenhd »

newpavlov wrote: post #3

So Varlink requires a proper specification of message types, but then uses god damn JSON to serialize messages? Why would you do that? Apparently, we don't have enough wasted cycles in modern software, just add a bunch more in a fundamental IPC infrastructure.

Not a fan of JSON per se for this type of messaging, but I don't think binary parser+validator combinations are that much faster. Plus, this forces C(++) developers to actually validate the data they're exchanging rather than casting raw structs from data buffers, which solves a huge vulnerability. Then again, the lack of 64 bit integers makes the protocol design comically unsuited.

I don't think the serialization/deserialisation overhead matters much when also taking into account the bus connection overhead. I would've preferred a better data exchange format but out of all problems I have with Linux IPC, the performance of JSON is the least of my worries.

Post reply on HN