Live data from Hacker News

Varlink – IPC to replace D-Bus gradually in systemd

varlink.org

91–100 of 273 posts

Re: Varlink – IPC to replace D-Bus gradually in systemd

#91
post #18

Earlier quoted context omitted.

To be honest I don't see why they don't use the existing IPC mechanism created for Android - binder and ashmem. It has been tested and developed in real world scenarios for over a decade by now.

Varlink was specifically designed to be available as early as possible when the kernel is booting and requires virtually nothing apart from good old sockets. But yes, definitely getting yet another one vibe from this. Then again, Dbus is not really what I would call an incredible piece of software - we are very much in the adequate territory if even - so anything shaking up the status quo can’t be too bad.

[deleted]

Re: Varlink – IPC to replace D-Bus gradually in systemd

#92
post #74

Earlier quoted context omitted.

I’m having trouble following what you mean. Yes, you can certainly send a whole SQLite database using any binary communication protocol as a way to exchange data today. You can even compress it before if you feel like it. It will not make for a good ipc format because, well, it’s not designed to be one but it would most definitely work. What’s your point?

There is this one somewhat widely deployed niche system that uses sqlite databases as RPC serialization format. The original idea behind that was limited virtual address space of the CE-based embedded devices, but the design got stuck.

Could you name it?

Re: Varlink – IPC to replace D-Bus gradually in systemd

#93

Earlier quoted context omitted.

CBOR is the obvious choice here. I've suggested it before and the systemd folks didn't seem completely opposed to it. Also because cbor parser is already in the dependencies due to FIDO2 disk unlocking

I genuinely dislike CBOR. Formats which require me to calculate the length of the message before sending it lead to bad library design which often leads to easily compromised code. Add in an "indefinite length" option and you've got potentially unbounded client memory usage to watch out for. As if that wasn't enough you get extensible tags so the meaning of any message is entirely dependent on the context it was sent…

"lead to bad library design"

how?

Re: Varlink – IPC to replace D-Bus gradually in systemd

#94
post #90
post #38

Earlier quoted context omitted.

Sigh. Varlink is designed for IPC, so the expected data rate is maybe hundreds of messages per second in the worst case. JSON can be parsed at 3 Gigabytes/second [0]. Even unoptimized stdlib parsers in scripting languages get 50 MB/sec. That's more than enough, standard library support is much more important for a project like this. And if there is ever a reason to send tar archive or video data, there is always "upg…

Hundreds of messages per second sounds like the performance I would expect from a 1980s computer. Why so incredibly slow?

Read the comment you’re replying to.

In it OP says it’s not slow, and the “hundreds of messages per second” you are referring to is not regarding the performance but the expected rate.

It’s possible for something to be fast, but only happen infrequently.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#95
post #69
post #30

Earlier quoted context omitted.

Note within the domain of this problem was the point. Which means on the same machine, with the same architecture and both ends being C which is what the init system is written in. You are adding more problems that don't exist to the specification. As for strings, just shove a char[4096] in there. Use a bit of memory to save a lot of parsing.

> As for strings, just shove a char[4096] in there. For the love of God, use a proper slice/fat pointer, please. Switching over to slices eliminates 90%+ of the issues with using C. Carrying around the base and the length eliminates a huge number of the overrun issues (especially if you don't store them consecutively). Splitting the base and the offset gives a huge amount of semantic information and makes serializati…

Broadly, I agree with you. C strings were a mistake. The standard library is full of broken shit that nobody should use, and worse, due to the myriad of slightly different "safe" string library functions, (a completely different subset of which is supported on any given platform,) which all have different edge cases, many people are over-confident that their C string code is actually correct. But is it? Is it checking errors? Does your function ensure that the destination buffer is null-terminated when it fails? Are you sure you don't have any off-by-one issues anywhere?

Correct as you may be though the argument here is that you should just write raw structs into Unix sockets. In this case you can't really use pointers. So, realistically, no slices either. In this context a fixed-size buffer is quite literally the only sensible thing you can do, but also, I think it's a great demonstration of why you absolutely shouldn't do this.

That said, if we're willing to get rid of the constraint of using only one plain C struct, you could use offsets instead of pointers. Allocate some contiguous chunk of memory for your entire request, place struct/strings/etc. in it, and use relative offsets. Then on the receiver side you just need some fairly basic validation checks to ensure none of the offsets go out of bounds. But at that point, you've basically invented part of Cap'n'proto, which begs the question... Why not just use something like that instead. It's pretty much the entire reason they were invented.

Oh well. Unfortunately the unforced errors of D-Bus seem like they will lead to an overcorrection in the other direction, turning the core of our operating system into something that I suspect nobody will love in the long term.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#96

So Varlink requires a proper specification of message types, but then uses god damn JSON to serialize messages? Why would you do that? Apparently, we don't have enough wasted cycles in modern software, just add a bunch more in a fundamental IPC infrastructure.

Lennart pointed out the fact you can see readable messages via strace to be a benefit of json. If their average message size is small and they don't expect to ever need high volume or large messages, then it's not really likely to be a problem in practice. He also pointed out that they waste far more cycles today on context switches (he suggested something on the order of 6 per IPC message).

Even if they eventually come to the conclusion they need something more performant in the future, it's probably still a net win to make the switch today. It's also possible it turns out fine for their use case.

I personally would think they would want to benefit from an IPC system that includes provisions for sending file descriptors.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#97
>reverse-domain name

Why do people keep replicating this terrible design decision? It puts the lowest-entropy part of the name at the beginning, so that you have to parse (or type) the maximum number of characters before you get a unique specification. Try tab-completing a "flatpak run" command sometime.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#98
post #30
post #29

Earlier quoted context omitted.

C structs are a terrible serialization format, since they are not a serialization format at all. Nothing guarantees that you will get consistent struct behavior on the same machine, but also, it only really solves the problem for C. For everything else, you have to duplicate the C structure exactly, including however it may vary per architecture (e.g. due to alignment.) And OK fine. It's not that bad, most C ABIs are…

Note within the domain of this problem was the point. Which means on the same machine, with the same architecture and both ends being C which is what the init system is written in. You are adding more problems that don't exist to the specification. As for strings, just shove a char[4096] in there. Use a bit of memory to save a lot of parsing.

Even being run on the same machine doesn't guarantee two independent processes agree on C struct layout compiled from the same source. For one, you could have something as simple as one compiled for 32bit, one 64, but even then compiler flags can impact struct layout.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#99
post #93

Earlier quoted context omitted.

I genuinely dislike CBOR. Formats which require me to calculate the length of the message before sending it lead to bad library design which often leads to easily compromised code. Add in an "indefinite length" option and you've got potentially unbounded client memory usage to watch out for. As if that wasn't enough you get extensible tags so the meaning of any message is entirely dependent on the context it was sent…

"lead to bad library design" how?

More complex buffer management in non-garbage collected languages.

man 3 cmsg

For a glimpse into the darker corners of hell. At least you get to grip it there. In GCed languages you have to hope the library designer gave you reasonable controls for timeouts, aborting large data transfers, and releasing buffers efficiently in a bursty network environment.

See all the controls any HTTP server gives you for this. "Maximum header size", "Maximum header(s) size", "Maximum Body Size", "Request Header Timeout", "Total Request Timeout."

None of those were added by default or by tasteful library authors. They were all added to answer to specific emergent security vulnerabilities that were discovered and then were demanded by the users.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#100
post #38

So Varlink requires a proper specification of message types, but then uses god damn JSON to serialize messages? Why would you do that? Apparently, we don't have enough wasted cycles in modern software, just add a bunch more in a fundamental IPC infrastructure.

Sigh. Varlink is designed for IPC, so the expected data rate is maybe hundreds of messages per second in the worst case. JSON can be parsed at 3 Gigabytes/second [0]. Even unoptimized stdlib parsers in scripting languages get 50 MB/sec. That's more than enough, standard library support is much more important for a project like this. And if there is ever a reason to send tar archive or video data, there is always "upg…

It doesn't matter what the rate is.

Of all the things they could do, they decided to choose one of the worst possible formats for IPC data[1], maybe to pander to the web crowd.

[1] Maybe XML would be a close competitor.

Post reply on HN