Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Varlink – IPC to replace D-Bus gradually in systemd

varlink.org

Post reply on HN
Print view View original

81–90 of 273 posts

Re: Varlink – IPC to replace D-Bus gradually in systemd

#81

Post by rkeene2 »

arianvanp wrote: post #25
jchw wrote: post #5

Earlier quoted context omitted.

Unfortunately I concur. There are a lot of problems with D-Bus, but that it wasn't using JSON wasn't one of them. The tooling for D-Bus is terrible. It has composite types but they are very primitive, e.g. you can't name fields in a structure without extensions (there's like three different extensions for this.) A lot of the code generators can't actually handle real-world schemas. Now I understand that D-Bus itself…

CBOR is the obvious choice here. I've suggested it before and the systemd folks didn't seem completely opposed to it. Also because cbor parser is already in the dependencies due to FIDO2 disk unlocking

Or an ASN.1 DER encoded sequence -- ASN.1 is more common than CBOR (being in a lot of PKI stuff) and there are already generators for DER (or JSON or XML...) for a given schema.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#82

Post by rkeene2 »

akira2501 wrote: post #60
arianvanp wrote: post #25

Earlier quoted context omitted.

CBOR is the obvious choice here. I've suggested it before and the systemd folks didn't seem completely opposed to it. Also because cbor parser is already in the dependencies due to FIDO2 disk unlocking

I genuinely dislike CBOR. Formats which require me to calculate the length of the message before sending it lead to bad library design which often leads to easily compromised code. Add in an "indefinite length" option and you've got potentially unbounded client memory usage to watch out for. As if that wasn't enough you get extensible tags so the meaning of any message is entirely dependent on the context it was sent…

FWIW, ASN.1 BER supports an indefinite length encoding (though DER does not; ASN.1 JSON and ASN.1 XML do as well)

Re: Varlink – IPC to replace D-Bus gradually in systemd

#83

Post by skissane »

capitainenemo wrote: post #72
IshKebab wrote: post #57

Earlier quoted context omitted.

That's not true. The number type in JSON is an arbitrary sized number. You're thinking of JavaScript which is not the same thing.

https://datatracker.ietf.org/doc/html/rfc7493#section-2.2 No. (which expands on the slightly fluffier text in https://datatracker.ietf.org/doc/html/rfc7159#section-6 which broadly says the same thing... and https://datatracker.ietf.org/doc/html/rfc8259#section-6 ) If you look at a parser matrix, flouting this rule is asking for trouble. Which is why the recommendations on MDN for bigint recommend an explicit parsing…

You are mixing up two different standards here, JSON and I-JSON. I-JSON is a subset of JSON with stricter requirements.

JSON recommends (for interoperability) that numbers be limited to those representable as IEEE 64-bit binary floats-but does not require that. A document which ignores that recommendation is still officially valid JSON. By contrast, I-JSON, as a subset of JSON, upgrades that recommendation to a requirement, so documents which ignore it are not valid I-JSON

Re: Varlink – IPC to replace D-Bus gradually in systemd

#84

Post by RandomThoughts3 »

dfox wrote: post #66
RandomThoughts3 wrote: post #44

Earlier quoted context omitted.

Structs are a part of C semantic. They are not an ipc format. You can somewhat use them like one if you take a lot of precaution about how they are laid out in memory including padding and packing but it’s very brittle. Asn.1 is both quite complicated and not very efficient. They could certainly have gone with protobufs or another binary serialisation format but would it really be better than the current choice? I do…

ASN.1 BER/DER is more or less the same thing as CBOR. The perceived complexity of ASN.1 comes from the schema language and specifications written in the convoluted telco/ITU-T style (and well, the 80's type system that has ~8 times two different types for “human readable string”).

> The perceived complexity of ASN.1 comes from the schema language and specifications written in the convoluted telco/ITU-T style (and well, the 80's type system that has ~8 times two different types for “human readable string”).

I can’t resist pointing that it’s basically a longer way of saying quite complicated and not very efficient.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#85

Post by capitainenemo »

skissane wrote: post #83
capitainenemo wrote: post #72

Earlier quoted context omitted.

https://datatracker.ietf.org/doc/html/rfc7493#section-2.2 No. (which expands on the slightly fluffier text in https://datatracker.ietf.org/doc/html/rfc7159#section-6 which broadly says the same thing... and https://datatracker.ietf.org/doc/html/rfc8259#section-6 ) If you look at a parser matrix, flouting this rule is asking for trouble. Which is why the recommendations on MDN for bigint recommend an explicit parsing…

You are mixing up two different standards here, JSON and I-JSON. I-JSON is a subset of JSON with stricter requirements. JSON recommends (for interoperability) that numbers be limited to those representable as IEEE 64-bit binary floats-but does not require that. A document which ignores that recommendation is still officially valid JSON. By contrast, I-JSON, as a subset of JSON, upgrades that recommendation to a requi…

Yes, they are different standards, but the purpose of I-JSON was specifically to solve interoperability problems. Those interoperability problems are due to the looseness of the original spec.

The others do not forbid larger numbers but note that doing that will make for bad times, and if you look at JSON parsing matrices (say, the "Parsing JSON is a Minefield" one), you can see that the majority of implementations do not do allow larger. So. float64 is a defacto standard and I-JSON is simply clarifying this. Given the main purpose of JSON is an interoperable data exchange format it would be a very bad idea to do otherwise.

https://en.wikipedia.org/wiki/JSON?useskin=vector#Interopera... (essentially same point made here)

Re: Varlink – IPC to replace D-Bus gradually in systemd

#86

Post by djbusby »

deivid wrote: post #9
newpavlov wrote: post #3

So Varlink requires a proper specification of message types, but then uses god damn JSON to serialize messages? Why would you do that? Apparently, we don't have enough wasted cycles in modern software, just add a bunch more in a fundamental IPC infrastructure.

There was a Varlink talk[0] a few days ago at All Systems Go, in that talk, Lennart mentioned that JSON is unfortunate (primarily due to no 64 bit ints) but it has the surprising benefit of being able to understand the bus messages when using `strace` for debugging. [0]: https://www.youtube.com/watch?v=emaVH2oWs2Y&list=PLWYdJViL9E...

Cool, readable messages in strace but still some odd binary log format?

Re: Varlink – IPC to replace D-Bus gradually in systemd

#87

Post by XorNot »

theamk wrote: post #39
arianvanp wrote: post #25

Earlier quoted context omitted.

CBOR is the obvious choice here. I've suggested it before and the systemd folks didn't seem completely opposed to it. Also because cbor parser is already in the dependencies due to FIDO2 disk unlocking

What is the data rate (messages, bytes) that you expect for this "D-Bus replacement protocol"? Which fraction of CPU will JSON ser/des will take to justify using CBOR?

I mean conversely every second week on HN someone complains about "inefficient modern development practices" and this is the sort of minor thing which would be trotted out as an example.

So I'd argue it's not an unreasonable question (although I lean closer in: better types then JSON is the problem, since a decent text serialization format is always going to be needed for debugging and development).

Re: Varlink – IPC to replace D-Bus gradually in systemd

#88

Post by akira2501 »

rkeene2 wrote: post #82
akira2501 wrote: post #60

Earlier quoted context omitted.

I genuinely dislike CBOR. Formats which require me to calculate the length of the message before sending it lead to bad library design which often leads to easily compromised code. Add in an "indefinite length" option and you've got potentially unbounded client memory usage to watch out for. As if that wasn't enough you get extensible tags so the meaning of any message is entirely dependent on the context it was sent…

FWIW, ASN.1 BER supports an indefinite length encoding (though DER does not; ASN.1 JSON and ASN.1 XML do as well)

ASN suffers from some of the other problems regardless of the encoding.

I was thinking about this more and I think FTP actually had a lot of the right ideas here. You want two channels. One for fast interactive command messaging and a second one coordinate especially just for bulk transfers arranged over the command channel. The design flaw in FTP of putting these on two separate ports put it immediately in conflict with firewall best practices so I think it went mostly unnoticed that it fundamentally is a good arrangement.

What you want is one channel, sequenced packets, with a defined maximum message length that is negotiated at startup and never changes during the life of the channel. This should probably never be more than 65k. There should be a known length packet type, and an unknown length packet type, with any attempt to send more than the negotiated maximum triggering an error and disconnect.

If you do need to send more than the negotiated amount you should open a new connection, in a bulk transfer mode, that after the initial handshake, has no protocol and is merely an associated stream of bytes that are wholly uninterpreted by the middleware layer other than to help you associate it with the sequenced packet that requested its initiation.

You'd actually be able to use TCP (with DSCP even), mostly avoid head of line blocking and multiplexer latencies, and have a reasonable security guarantee in the sequenced packet mode, and never have a protocol which pretends that 4GB strings in the middle of a packet are necessary or even a good idea to "support."

The downfall of this is that it would be much harder to implement it on a serverless architecture and would be nearly as complicated as a protocol as WebSocket ends up being. It might be worth playing with as a concept anyways.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#89

Post by WhyNotHugo »

nly wrote: post #73

Why don't they just adopt the same fundamental IPC design as Wayland? Swap out the root wl_display singleton object for something dbus appropriate and away you go? More or less all the display specific stuff is an extension on top of the core design. https://wayland-book.com/protocol-design.html

I often wonder why that protocol isn’t used in other fields. It has clear schemas with documentation, and existing codegen implementations.

OTOH, it’s too complex for simple protocols with a few messages.

IIRC, Pipewire uses a variation of the Wayland protocol.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#90

Post by fanf2 »

theamk wrote: post #38
newpavlov wrote: post #3

So Varlink requires a proper specification of message types, but then uses god damn JSON to serialize messages? Why would you do that? Apparently, we don't have enough wasted cycles in modern software, just add a bunch more in a fundamental IPC infrastructure.

Sigh. Varlink is designed for IPC, so the expected data rate is maybe hundreds of messages per second in the worst case. JSON can be parsed at 3 Gigabytes/second [0]. Even unoptimized stdlib parsers in scripting languages get 50 MB/sec. That's more than enough, standard library support is much more important for a project like this. And if there is ever a reason to send tar archive or video data, there is always "upg…

Hundreds of messages per second sounds like the performance I would expect from a 1980s computer. Why so incredibly slow?
Post reply on HN