Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Varlink – IPC to replace D-Bus gradually in systemd

varlink.org

Post reply on HN
Print view View original

261–270 of 273 posts

Re: Varlink – IPC to replace D-Bus gradually in systemd

#261

Post by jchw »

otabdeveloper4 wrote: post #255
jchw wrote: post #22

Earlier quoted context omitted.

I don't know how to tell you this, but, you don't need to implement an RPC protocol in bash, nor do you need FFI. You can use CLI tools like `dbus-send`. I pray to God nothing meaningful is actually doing what you are insinuating in any serious environment.

I'm trying to tell you that something that isn't straceable and greappable shouldn't belong in your system services stack.

Strace/ptrace is awful, and I don't know what "greappable" means here. Nothing stops us from adding DTrace probes.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#262

Post by Netch »

btreecat wrote: post #188
quotemstr wrote: post #108

Christ almighty, I have seldom seen a technical proposal as bad as this one. I want off Mr. Poettering's wild ride. No 64 bit integers? No file descriptor transport? No message sequence numbers? Just cut it out already. Dbus doesn't do something? Extend dbus. Don't make some other random and worse things.

>No 64 bit integers? No file descriptor transport? No message sequence numbers? Are these required or useful in this context? >Dbus doesn't do something? Extend dbus. How are you so certain that this is the best use of resources, do you have personal XP in the problem domain? > Don't make some other random and worse things. How did you determine that these changes are "worse" or "random?"

> Are these required or useful in this context?

At least 64-bit numbers are crucial in any system context.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#263

Post by Netch »

Vegemeister wrote: post #97

>reverse-domain name Why do people keep replicating this terrible design decision? It puts the lowest-entropy part of the name at the beginning, so that you have to parse (or type) the maximum number of characters before you get a unique specification. Try tab-completing a "flatpak run" command sometime.

If you mention tab-completing, if starting with the detailest component it wonʼt work at all because of context search problems. If you press something like "alice", in deepest-first order it will have to search the whole tree which could contain millions of entities. Worse than, some subtrees may be dynamically loaded during the completion request.

To mitigate senseless tree top listing, IDEs propose a bunch of means like context-related hints or unpacking forms like "o.e.t" to "org.example.test".

Re: Varlink – IPC to replace D-Bus gradually in systemd

#264

Post by Netch »

akira2501 wrote: post #60
arianvanp wrote: post #25

Earlier quoted context omitted.

CBOR is the obvious choice here. I've suggested it before and the systemd folks didn't seem completely opposed to it. Also because cbor parser is already in the dependencies due to FIDO2 disk unlocking

I genuinely dislike CBOR. Formats which require me to calculate the length of the message before sending it lead to bad library design which often leads to easily compromised code. Add in an "indefinite length" option and you've got potentially unbounded client memory usage to watch out for. As if that wasn't enough you get extensible tags so the meaning of any message is entirely dependent on the context it was sent…

> I genuinely dislike CBOR. Formats which require me to calculate the length of the message before sending it

CBOR is not. No full message length. Only primitive type values. Compound type values require length _in items_ (not bytes) but indefinite encoding is possible if allowed ("deterministic" constraint to message forbids it).

So technically this doesnʼt differ from e.g. JSON where you have to wait until final "}" or "]" for a long compound item, and have to install an artificial limit.

> Add in an "indefinite length" option and you've got potentially unbounded client memory usage to watch out for. > As if that wasn't enough you get extensible tags so the meaning of any message is entirely dependent on the context it was sent in.

So does any such a protocol. JSON, XML, any ASN.1, CBOR, whatever. If there are no extensible tags, there are field names - or they will be added by customer using what is allowed. Iʼve seen this in ASN.1 sequence-of pairs of name+value, directly emulating JSON-like dictionary. To limit it is not how security issues are handled.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#265

Post by Netch »

cryptonector wrote: post #105
RandomThoughts3 wrote: post #44

Earlier quoted context omitted.

Structs are a part of C semantic. They are not an ipc format. You can somewhat use them like one if you take a lot of precaution about how they are laid out in memory including padding and packing but it’s very brittle. Asn.1 is both quite complicated and not very efficient. They could certainly have gone with protobufs or another binary serialisation format but would it really be better than the current choice? I do…

> Structs are a part of C semantic. Uh, no, structs, records, whatever you want to call them, are in many, if not most programming languages. "Structs" is not just "C structs" -- it's just shorthand for "structured data types" (same as in C!). > Asn.1 is both quite complicated and not very efficient. Every rich encoding system is complicated. As for efficiency, ASN.1 has many encoding rules, some of which are quite b…

It seems you posit taglessness to be a universal crucial merit of any encoding scheme. This is good in an ideal world, heh.

I have had a misfortune to work for 5G which is full of PER-encoded protocols. Dealing with discrepancies in them - incompatible changes in 3GPP standard versions, different vendorsʼ errors, combined with usually low level of developers and managers in a typical corporation - was an utter nightmare.

IETF, in general, provides a good policy combining truly fixed binary protocols when they are unavoidable (IP/TCP/UDP levels) and flexible, often text, protocols where there is no substantial overhead from their use. Their early moves, well, suffered from over-grammaticalization (as RFC822). CBOR is nice here because it combines tagness and compactness. 3-bit basic tag combined with value (if fit) or length, it is commensurable with OER in efficiency but is decodable without scheme - and it is extremely useful in practice.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#266

Post by Netch »

cryptonector wrote: post #209
RandomThoughts3 wrote: post #149

Earlier quoted context omitted.

> That's very wrong. ASN.1 is complicated because it's quite complete by comparison to other syntaxes So, it's quite complicated. Yes, what I have been saying from the start. If you start the conversation by "you can define a small subset of this terrible piece of technology which is bearable", it's going to be hard convincing people it's a good idea. > Cavalier attitudes like "ASN.1 is too complicated" lead to bad r…

> So, it's quite complicated. Subsets of ASN.1 that match the functionality of Protocol Buffers are not "quite complicated" -- they are no more complicated than PB. > Still, ASN.1 is a telco protocol through and through. Not really. The ITU-T developed it, so it gets used a lot in telco protocols, but the IETF also makes a lot of use of it. It's just a syntax and set of encoding rules. And so what if it were "a telco…

> The tooling? There's open source tooling that generates code like any XDR tooling and like PB tooling and like MSFT RPC tooling.

There is no open source tooling that combines really used scheme understanding - in 5G this includes parameterized specifications by X.683 - and decoder able to show partially decoded message before an error, with per-bit explanation of rules led to its encoding.

> E.g., PKI uses DER, so a TLV encoding, thus it's inefficient.

When it is used, ~5% space economy is never worth people efforts to diagnose any problem. I strictly vote for this "inefficiency".

Re: Varlink – IPC to replace D-Bus gradually in systemd

#267

Post by Netch »

poettering wrote: post #160
jeroenhd wrote: post #150

Earlier quoted context omitted.

Not a fan of JSON per se for this type of messaging, but I don't think binary parser+validator combinations are that much faster. Plus, this forces C(++) developers to actually validate the data they're exchanging rather than casting raw structs from data buffers, which solves a huge vulnerability. Then again, the lack of 64 bit integers makes the protocol design comically unsuited. I don't think the serialization/de…

The 64bit issue is certainly an issue, but very much overblown. First of all, in systemd, which is a heavy D-Bus user, we effectively IRL only send integers > 2^53 when we use UINT64_MAX as a special "niche" marker meaning "unlimited"/"unset"/"undefined". But the thing is that JSON has a proper concept for this: the "null" value (or simply not specifying a specific JSON field). Hence, in reasonably clean APIs this is…

> if you do what the various specs on the internet suggest you do if you have an integer > 2^53: you encode it as decimal value as a string.

Hexadecimal is better - more compact, much less encoding/decoding effort. Alternatively, array of two integers does as well.

Text forms for any error, not NaN, are also useful.

But issues with non-UTF-8 encodings - as in CJK world - may spoil the picture pretty more.

Re: Varlink – IPC to replace D-Bus gradually in systemd

#268

Post by Netch »

accelbred wrote: post #249
poettering wrote: post #160

Earlier quoted context omitted.

The 64bit issue is certainly an issue, but very much overblown. First of all, in systemd, which is a heavy D-Bus user, we effectively IRL only send integers > 2^53 when we use UINT64_MAX as a special "niche" marker meaning "unlimited"/"unset"/"undefined". But the thing is that JSON has a proper concept for this: the "null" value (or simply not specifying a specific JSON field). Hence, in reasonably clean APIs this is…

Are strings used for timestamps? Those are usually why I'm sending 64-bit ints over IPC.

If you want more than 1-second precision, 64 bits are not enough. (Hmm does all C++ std::chrono implementation utilize 128-bit integers for nanosecond precision?)

Re: Varlink – IPC to replace D-Bus gradually in systemd

#269

Post by Netch »

jchw wrote: post #184
jcelerier wrote: post #175

Earlier quoted context omitted.

It's not that simple: [1.2,1.3,1.4] is 13 bytes, the equivalent binary storage is 3*sizeof(double) (24 bytes) + whatever overhead exists for representing a dynamic array in your language

Personal opinion: don't put floating point numbers into an API. If you really want real numbers passing them as decimal string is actually sensible. This can be done without using JSON, of course. Can also do fixed precision as well, which will store reasonably well into a VLQ. OTOH this is pretty cherry picked. It is questionable if you really need floating point numbers if your numbers are all this small and low pr…

> If you really want real numbers passing them as decimal string is actually sensible.

Hexadecimal, if so. Nearly any decent modern language and runtime can it.

    $ python
    >>> import math
    >>> math.pi.hex()
    '0x1.921fb54442d18p+1'
This gives exact value without any rounding effects and conversion cost.

(I donʼt count decimal floating-point as it is quite rare and its application is faint. In overwhelming most of cases, fixed point is better.)

Re: Varlink – IPC to replace D-Bus gradually in systemd

#270

Post by Netch »

IgorPartola wrote: post #124
akira2501 wrote: post #88

Earlier quoted context omitted.

ASN suffers from some of the other problems regardless of the encoding. I was thinking about this more and I think FTP actually had a lot of the right ideas here. You want two channels. One for fast interactive command messaging and a second one coordinate especially just for bulk transfers arranged over the command channel. The design flaw in FTP of putting these on two separate ports put it immediately in conflict…

You would need a pretty intense QoS system built for this if you are using something like TCP for this using two different connections. Underneath it would still use IPv4/6 packets and you only have so much bandwidth. The bulk transfer channel might stall the transfer of control packets temporarily. You would need to tightly control and multiplex the messages yourself, needing to do this in one connection or using so…

> SCTP

> It also has multiplexing built right in.

It is not. You canʼt stop, for example, receiving from stream 0 during getting high-priority data from stream 9. No userlevel API allows to specify "now receive a message exactly from stream 9". Data packet TSNs are the same number sequence for all streams. You canʼt normally ignore a single data TSN for a stream payload: you have to issue SACK and brook constant retransmissions from other side. You canʼt specify receive window separately for each stream, to calm the sender down while the streamʼs receive buffer is full.

I donʼt know who and why spreads the myth it is really multiplexing, but now these stream numbers are merely another type of per-message external tag. All other is merely scam. It seems that initial design was moving toward this possibility but then something unfortunate happened.

> This means you don’t have to devise message length communication into your application layer protocol and can rely on your transport.

This is, well, tasty from it.

Post reply on HN