Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Varlink – IPC to replace D-Bus gradually in systemd

varlink.org

Post reply on HN
Print view View original

151–160 of 273 posts

Re: Varlink – IPC to replace D-Bus gradually in systemd

#151

Post by dathinab »

capitainenemo wrote: post #37
gmokki wrote: post #17

Earlier quoted context omitted.

I do not understand where the 64bit integers not working with JSON comes from. JSON the format had no limit on integer size. And all Java JSON libraries I know can handle arbitrary prevsion integers (BigInt) and 32/64bit int/long types when serializing and deserializing. Quick googling shows that also JavaScript has proper support for 64bit integers with their BigInt type, and it can be used to deserialize incoming d…

The number type in JSON is 64 bit float, limiting integers without loss of precision to 2⁵³-1. BigInt is a new concept and not technically supported. So whether it works in your random library of choice is probably a potshoot. "Use within JSON: Using JSON.stringify() with any BigInt value will raise a TypeError, as BigInt values aren't serialized in JSON by default. " https://developer.mozilla.org/en-US/docs/Web/Java…

yes but no

there is more then one JSON standard and while the JS in JSON stands for JavaScript it doesn't directly means it is using JS types as confusing as it might seem.

the initial definition of JSON by itself has no limit on integer size at all, JSON with number which can only be represented with BigInt _totally is a thing_ just not so much in the Js ecosystem. Furthermore systems accidental expecting full i64 or u64 number ranges are as much a thing, and this leading to subtle bugs the moment a different (de-)serializer is used is another issues.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#152

Post by gpderetta »

poettering wrote: post #125
newpavlov wrote: post #3

So Varlink requires a proper specification of message types, but then uses god damn JSON to serialize messages? Why would you do that? Apparently, we don't have enough wasted cycles in modern software, just add a bunch more in a fundamental IPC infrastructure.

The marshalling cost for JSON is negligible. Yes, it might be a bit slower than GVariant for example, but only by some fractional linear factor. And on small messages (which D-Bus currently always is, due to message size constraints enforced by broker) the difference is impossible to measure. To a point it really doesn't matter, in particular as JSON parsers have been ridiculously well optimized in this world. What d…

>With a simple "strace" I can now reasonably trace my programs, which a binary serialization will never allow you

Doesn't systemd use binary logging?

Re: Varlink – IPC to replace D-Bus gradually in systemd

#153

Post by guappa »

xorcist wrote: post #135
poettering wrote: post #125

Earlier quoted context omitted.

The marshalling cost for JSON is negligible. Yes, it might be a bit slower than GVariant for example, but only by some fractional linear factor. And on small messages (which D-Bus currently always is, due to message size constraints enforced by broker) the difference is impossible to measure. To a point it really doesn't matter, in particular as JSON parsers have been ridiculously well optimized in this world. What d…

But .. how can anyone use strace? That's not JSON. And serialization is cheap !

Do you know what strace is?

It's a command that prints the system calls (done via library). So you see write(55, "[1,2,3,4]\n")

Re: Varlink – IPC to replace D-Bus gradually in systemd

#154

Post by guappa »

Dunedan wrote: post #141
solarengineer wrote: post #134

Earlier quoted context omitted.

One can view the binary log files using journalctl. Per https://systemd.io/JOURNAL_FILE_FORMAT/ , the benefits of the binary format are: The systemd journal stores log data in a binary format with several features: Fully indexed by all fields Can store binary data, up to 2^64-1 in size Seekable Primarily append-based, hence robust to corruption Support for in-line compression Support for in-line Forward Secure Sealin…

> Primarily append-based, hence robust to corruption It's so robust, it doesn't even let you modify the journal if you want to (e.g. https://github.com/systemd/systemd/issues/20673 ). > Support for in-line compression Mind that journald only supports compressing single lines, but not the whole journal ( https://github.com/systemd/systemd/issues/31358 ), which is pretty limiting.

In sicily we'd call this "vuliri a vutti china e a mugghieri 'mbriaca".

It's a tradeoff, if you do full compression clearly it won't be fast.

You're free to compress it again before archiving it.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#155

Post by gpderetta »

AshamedCaptain wrote: post #75
altfredd wrote: post #67

Earlier quoted context omitted.

> WTF? Sure you can do on top of even pipes. Even XDR could... Of course, you "can". Implement message-based communication on top of streaming. Emulate blocking calls on top of non-blocking socket API. Implement authentication via ancillary data (try not to throw up in process). Use some clever tricks to solve priority inversion. Somehow distinguish your fds from all others file descriptors to ensure that no confused…

> Of course, you "can". What is your point here? You claimed that "other IPC solutions" _could not_ do this on top of sockets, but as I've shown and now you admit, you definitely _can_. Obviously I'm not suggesting you roll this by hand, nor claiming that a plain UNIX pipe is a fully featured RPC system; just that there's a million RPC systems out there that do it and don't need anything else than sockets.

I don't know what binder does exactly, but rendez-vous synchronization is surprisingly hard to do efficiently on POSIX systems without excessive context switches.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#156

Post by RandomThoughts3 »

cryptonector wrote: post #105
RandomThoughts3 wrote: post #44

Earlier quoted context omitted.

Structs are a part of C semantic. They are not an ipc format. You can somewhat use them like one if you take a lot of precaution about how they are laid out in memory including padding and packing but it’s very brittle. Asn.1 is both quite complicated and not very efficient. They could certainly have gone with protobufs or another binary serialisation format but would it really be better than the current choice? I do…

> Structs are a part of C semantic. Uh, no, structs, records, whatever you want to call them, are in many, if not most programming languages. "Structs" is not just "C structs" -- it's just shorthand for "structured data types" (same as in C!). > Asn.1 is both quite complicated and not very efficient. Every rich encoding system is complicated. As for efficiency, ASN.1 has many encoding rules, some of which are quite b…

> Uh, no, structs, records, whatever you want to call them

It's plenty clear from discussion context that OP is talking about C struct but yes, replace C with any languages which suit you. It will still be part of the language semantic and not an IPC specification.

The point is you can't generally use memory layout as an IPC protocol because you generally have no guarantee that it will be the same for all architectures.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#157

Post by poettering »

gpderetta wrote: post #152
poettering wrote: post #125

Earlier quoted context omitted.

The marshalling cost for JSON is negligible. Yes, it might be a bit slower than GVariant for example, but only by some fractional linear factor. And on small messages (which D-Bus currently always is, due to message size constraints enforced by broker) the difference is impossible to measure. To a point it really doesn't matter, in particular as JSON parsers have been ridiculously well optimized in this world. What d…

>With a simple "strace" I can now reasonably trace my programs, which a binary serialization will never allow you Doesn't systemd use binary logging?

Not really. We use two text based formats for logging: BSD syslog, and systemd's structured logging (which is basically an env block, i.e. a key-value set, with some tweaks). Programs generate text logs, journald reads text logs hence. Programs that read from the journal get the text-based key/value stuff back, usually.

(Yes, we then store the structure log data on disk in a binary format. Lookup indexes are just nasty in text based formats).

Hence, not sure what the Journal has to do with Varlink, but any IPC that the journal does is text-based, and very nicely strace'able in fact, I do that all the time.

[Maybe, when trying to be a smartass, try to be "smart", and not just an "ass"?]

Re: Varlink – IPC to replace D-Bus gradually in systemd

#158

Post by poettering »

steventhedev wrote: post #128
poettering wrote: post #125

Earlier quoted context omitted.

The marshalling cost for JSON is negligible. Yes, it might be a bit slower than GVariant for example, but only by some fractional linear factor. And on small messages (which D-Bus currently always is, due to message size constraints enforced by broker) the difference is impossible to measure. To a point it really doesn't matter, in particular as JSON parsers have been ridiculously well optimized in this world. What d…

The danger I see is that JSON has lots of edge behavior around deserialization, and some languages will deserialize a 100 digit number differently. If the main benefit is removing the broker and the need for rate limiting - it could have been accomplished without using JSON.

You are writing this as if JSON was a newly invented thing, and not a language that has become the lingua franca of the Internet when it comes to encoding structured data. Well understood, and universally handled, since 1997.

A 100 digit number cannot be encoded losslessly in D-Bus btw, nor in the far majority of IPC marshallings on this word.

Having done systems-level OS development since 25y or so I never felt the burning urge to send a 100 digit number over local IPC.

Not that 100 digit numbers aren't useful, even in IPC, but typically, that's a cryptography thing, and they generally use their own serializations anyway.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#159

Post by tarasglek »

All the the hate for json here, but at the bottom there is a bridge screenshot. Varlink is designed such that it can trivially be piped across different systems and be maximally interoperable due simple to parse and encode protocol. We might end up with some IPC interface that a lot me tooling can understand. Eg imagine that instead of some daemons supporting sighup to reload, they could also confirm that they reloaded, provided Prometheus style metrics etc

Re: Varlink – IPC to replace D-Bus gradually in systemd

#160

Post by poettering »

jeroenhd wrote: post #150
newpavlov wrote: post #3

So Varlink requires a proper specification of message types, but then uses god damn JSON to serialize messages? Why would you do that? Apparently, we don't have enough wasted cycles in modern software, just add a bunch more in a fundamental IPC infrastructure.

Not a fan of JSON per se for this type of messaging, but I don't think binary parser+validator combinations are that much faster. Plus, this forces C(++) developers to actually validate the data they're exchanging rather than casting raw structs from data buffers, which solves a huge vulnerability. Then again, the lack of 64 bit integers makes the protocol design comically unsuited. I don't think the serialization/de…

The 64bit issue is certainly an issue, but very much overblown.

First of all, in systemd, which is a heavy D-Bus user, we effectively IRL only send integers > 2^53 when we use UINT64_MAX as a special "niche" marker meaning "unlimited"/"unset"/"undefined". But the thing is that JSON has a proper concept for this: the "null" value (or simply not specifying a specific JSON field). Hence, in reasonably clean APIs this is mostly a non-issue.

That said, sd-varlink/sd-json (i.e. systemd's implementation of it), of course is 64bit signed and unsigned integer clean when it processes JSON. More-over it automatically handles if you do what the various specs on the internet suggest you do if you have an integer > 2^53: you encode it as decimal value as a string.

Would it be better if JSON would have been more precise on this, yes. Is it a big issue? No, not at all.

Post reply on HN