Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Varlink – IPC to replace D-Bus gradually in systemd

varlink.org

Post reply on HN
Print view View original

101–110 of 273 posts

Re: Varlink – IPC to replace D-Bus gradually in systemd

#101

Post by rkeene2 »

akira2501 wrote: post #88
rkeene2 wrote: post #82

Earlier quoted context omitted.

FWIW, ASN.1 BER supports an indefinite length encoding (though DER does not; ASN.1 JSON and ASN.1 XML do as well)

ASN suffers from some of the other problems regardless of the encoding. I was thinking about this more and I think FTP actually had a lot of the right ideas here. You want two channels. One for fast interactive command messaging and a second one coordinate especially just for bulk transfers arranged over the command channel. The design flaw in FTP of putting these on two separate ports put it immediately in conflict…

I think the protocol on which PDUs are exchanged on is a bit orthogonal to the PDU protocol itself here, but the split-mode networking protocols (FTP, IRC DCC, mosh, SIP, etc) always do end up getting less used because establishing two different connections is a bigger burden and there really needs to be a great advantage to doing so, and often require other considerations (better connection tracking, TURN/ICE servers, etc).

For a localhost/UNIX domain protocol, it might work since most of these considerations are significantly reduced.

As to DSCP, it might be useful across controlled networks but in general, in my experience it's not really ever honored broadly.

Back on the original topic, when I was building this for my Linux distribution I ended up just using basic JSON with TLS (for client certificate authentication) though since having authentication for remote management was a goal, and once I was already having to perform a TLS negotiation then the PDU consideration for performance wasn't something to really spend too much time on.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#103

Post by markasoftware »

surajrmal wrote: post #96
newpavlov wrote: post #3

So Varlink requires a proper specification of message types, but then uses god damn JSON to serialize messages? Why would you do that? Apparently, we don't have enough wasted cycles in modern software, just add a bunch more in a fundamental IPC infrastructure.

Lennart pointed out the fact you can see readable messages via strace to be a benefit of json. If their average message size is small and they don't expect to ever need high volume or large messages, then it's not really likely to be a problem in practice. He also pointed out that they waste far more cycles today on context switches (he suggested something on the order of 6 per IPC message). Even if they eventually c…

> Lennart pointed out the fact you can see readable messages via strace to be a benefit of json.

from the guy who brought you binary logfiles!

Re: Varlink – IPC to replace D-Bus gradually in systemd

#104

Post by [deleted] »

theamk wrote: post #38
newpavlov wrote: post #3

So Varlink requires a proper specification of message types, but then uses god damn JSON to serialize messages? Why would you do that? Apparently, we don't have enough wasted cycles in modern software, just add a bunch more in a fundamental IPC infrastructure.

Sigh. Varlink is designed for IPC, so the expected data rate is maybe hundreds of messages per second in the worst case. JSON can be parsed at 3 Gigabytes/second [0]. Even unoptimized stdlib parsers in scripting languages get 50 MB/sec. That's more than enough, standard library support is much more important for a project like this. And if there is ever a reason to send tar archive or video data, there is always "upg…

[deleted]

Re: Varlink – IPC to replace D-Bus gradually in systemd

#105

Post by cryptonector »

RandomThoughts3 wrote: post #44
nbbnbb wrote: post #23

Earlier quoted context omitted.

protobufs / asn.1 / structs ... Edit: hell even XML is better than this!

Structs are a part of C semantic. They are not an ipc format. You can somewhat use them like one if you take a lot of precaution about how they are laid out in memory including padding and packing but it’s very brittle. Asn.1 is both quite complicated and not very efficient. They could certainly have gone with protobufs or another binary serialisation format but would it really be better than the current choice? I do…

> Structs are a part of C semantic.

Uh, no, structs, records, whatever you want to call them, are in many, if not most programming languages. "Structs" is not just "C structs" -- it's just shorthand for "structured data types" (same as in C!).

> Asn.1 is both quite complicated and not very efficient.

Every rich encoding system is complicated. As for efficiency, ASN.1 has many encoding rules, some of which are quite bad (BER/DER/CER, which are the first family of ERs, and so many thing ASN.1 == BER/DER/CER, but that's not the case), and some of which are very efficient (PER, OER). Heck, you can use XML and JSON as ASN.1 encoding rules (XER, JER).

> They could certainly have gone with protobufs or another binary serialisation format but would it really be better than the current choice?

Protocol buffers is a tag-length-value (TLV) encoding, same as BER/DER/CER. Having to have a tag and length for every value encoded is very inefficient, both in terms of encoding size as well as in terms of computation.

The better ASN.1 ERs -PER and OER- are much more akin to XDR and Flat buffers than to protobufs.

> I don’t think the issue they are trying to solve is related to serialisation anyway. Seems to me they are unhappy about the bus part not the message format part.

This definitely seems to be be the case.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#106

Post by ahupp »

greatgib wrote: post #61

[flagged]

Is modularity an unalloyed good? Modularity comes with tradeoffs that mean you end up with, well, a bunch of discrete modules rather than something that works cohesively. That's why systemd was adopted pretty much everywhere, ignoring the arbitrary modular boundaries results in a more useful tool.

Some analogs: modern filesystems like ZFS and BTRFS that combine volume management with the filesystem, every service that's consciously chosen to deploy a monolith instead of microservices, and so on, every deployment that chooses to statically link, etc.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#107

Post by cryptonector »

dfox wrote: post #66
RandomThoughts3 wrote: post #44

Earlier quoted context omitted.

Structs are a part of C semantic. They are not an ipc format. You can somewhat use them like one if you take a lot of precaution about how they are laid out in memory including padding and packing but it’s very brittle. Asn.1 is both quite complicated and not very efficient. They could certainly have gone with protobufs or another binary serialisation format but would it really be better than the current choice? I do…

ASN.1 BER/DER is more or less the same thing as CBOR. The perceived complexity of ASN.1 comes from the schema language and specifications written in the convoluted telco/ITU-T style (and well, the 80's type system that has ~8 times two different types for “human readable string”).

That "convoluted telco/ITU-T style" yields amazingly high quality specifications. I'll take X.68x / X.69x any day over most Internet RFCs (and I've written a number of Internet RFCs). The ITU-T puts a great deal of effort into its specs, or at least the ASN.1 working group did.

ASN.1 is not that complicated. Pity that fools who thought ASN.1 was complicated re-invented the wheel quite poorly (Protocol Buffers I'm looking at you).

Re: Varlink – IPC to replace D-Bus gradually in systemd

#108

Post by quotemstr »

Christ almighty, I have seldom seen a technical proposal as bad as this one. I want off Mr. Poettering's wild ride. No 64 bit integers? No file descriptor transport? No message sequence numbers? Just cut it out already. Dbus doesn't do something? Extend dbus. Don't make some other random and worse things.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#109

Post by quotemstr »

jsnell wrote: post #12
newpavlov wrote: post #3

So Varlink requires a proper specification of message types, but then uses god damn JSON to serialize messages? Why would you do that? Apparently, we don't have enough wasted cycles in modern software, just add a bunch more in a fundamental IPC infrastructure.

I've done similar things in the past (json over udp-to-localhost, with a schema description used to generate the code for both parsing and generating on each end). It's a totally reasonable point in the design space, and I literally never saw a reason to revisit the decision to use json for that system. You'd do that because everything under the sun knows how to generate and parse json. Performance looks like a total…

The problem isn't JSON per se. Parsing JSON is fast. The problem is the simplistic interaction and data model that's just going to move complexity from universal IPC layers to ad hoc application logic that every person has to learn individually. Terrible.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#110

Post by quotemstr »

deivid wrote: post #9
newpavlov wrote: post #3

So Varlink requires a proper specification of message types, but then uses god damn JSON to serialize messages? Why would you do that? Apparently, we don't have enough wasted cycles in modern software, just add a bunch more in a fundamental IPC infrastructure.

There was a Varlink talk[0] a few days ago at All Systems Go, in that talk, Lennart mentioned that JSON is unfortunate (primarily due to no 64 bit ints) but it has the surprising benefit of being able to understand the bus messages when using `strace` for debugging. [0]: https://www.youtube.com/watch?v=emaVH2oWs2Y&list=PLWYdJViL9E...

Oh my God. I'm genuinely struggling to avoid using an endless stream of profanity here. If our problem is that our observability tools suck, the solution is to improve these tools, not mutilate our IPC mechanism to accommodate the limitations of these tools.

Christ almighty, this is a terrible proposal.

Post reply on HN