Bus1: a new Linux interprocess communication proposal • Hacker News Board

Re: Bus1: a new Linux interprocess communication proposal

#51

Post by lathiat » Thu, Aug 18, 2016, 11:38 AM UTC

IshKebab wrote: post #48

qwertyuiop924 wrote: post #45

No. Nonononononononono. We were ALMOST FREE! Just get these high-level abstractions out of my effing kernel. They don't belong there. Linus said as much himself. The only reason kdbus, this thing's predecessor, got so much as a glance is that Linus trusts Greg KH. A trust I would now consider somewhat misplaced. This is yet another piece of crap that was tacked on by systemd nonsense. It's non-portable, and they want…

I agree. I can't see any real reason for this to be in the kernel. Is it faster? Does it need access to memory in some way you can't do in userspace? The only practical reason I can think is if it is in the kernel you can 100% rely on it being present and available. Userspace processes can be killed. So perhaps the real solution is to provide some kind of mechanism for userspace processes that act like kernel drivers…

The two main reasons I know/care about (there may be more) are that it can be much faster (zero copy, reduced context switches) and also guarantee the message isn't changed while reading (by sealing the originating buffer).

The last is something you need kernel help to enforce.

Re: Bus1: a new Linux interprocess communication proposal

#52

Post by IshKebab » Thu, Aug 18, 2016, 11:38 AM UTC

chme wrote: post #24

Annatar wrote: post #16

Earlier quoted context omitted.

That's the thing about Linux: they have literally not one, but multiple armies of "programmers", and yet even after 20+ years of an absolutely furious development pace, basic things like inter-process communication still do not work desirably. Startup/shutdown? Yep, you know it already, it's still in flux (SMF in illumos has been humming along for a decade now). Filesystems? Flux. Observability? Flux. libc, what libc…

> Flux. And while all that is constantly changing, its still possible to compile very old user space software and run it in on a current kernel.

Yeah but is it possible to run very old binaries on a current kernel? Unlikely. What about binaries compiled on a newer system than your own? Even less likely.

Re: Bus1: a new Linux interprocess communication proposal

#53

Post by sjellis » Thu, Aug 18, 2016, 11:41 AM UTC

qwertyuiop924 wrote: post #50

IshKebab wrote: post #48

Earlier quoted context omitted.

I agree. I can't see any real reason for this to be in the kernel. Is it faster? Does it need access to memory in some way you can't do in userspace? The only practical reason I can think is if it is in the kernel you can 100% rely on it being present and available. Userspace processes can be killed. So perhaps the real solution is to provide some kind of mechanism for userspace processes that act like kernel drivers…

That's just a daemon. The reason this is in the kernel is that DBus is slow (according to Linus, because it could have been written better by monkeys with typewriters), and Lennart wanted it in the kernel/systemd, and Greg KH agreed, and Linus trusts Greg.

The announcement email from David Herrmann explains that they did consider a user-space implementation, and lists the reasons why they decided that it had to be in-kernel:

https://lists.linuxfoundation.org/pipermail/ksummit-discuss/...

Re: Bus1: a new Linux interprocess communication proposal

#54

Post by qwertyuiop924 » Thu, Aug 18, 2016, 11:47 AM UTC

sjellis wrote: post #53

qwertyuiop924 wrote: post #50

Earlier quoted context omitted.

That's just a daemon. The reason this is in the kernel is that DBus is slow (according to Linus, because it could have been written better by monkeys with typewriters), and Lennart wanted it in the kernel/systemd, and Greg KH agreed, and Linus trusts Greg.

The announcement email from David Herrmann explains that they did consider a user-space implementation, and lists the reasons why they decided that it had to be in-kernel: https://lists.linuxfoundation.org/pipermail/ksummit-discuss/...

I was talking about KDbus, the predecessor to this.

Re: Bus1: a new Linux interprocess communication proposal

#55

Post by qwertyuiop924 » Thu, Aug 18, 2016, 11:49 AM UTC

lathiat wrote: post #51

IshKebab wrote: post #48

Earlier quoted context omitted.

I agree. I can't see any real reason for this to be in the kernel. Is it faster? Does it need access to memory in some way you can't do in userspace? The only practical reason I can think is if it is in the kernel you can 100% rely on it being present and available. Userspace processes can be killed. So perhaps the real solution is to provide some kind of mechanism for userspace processes that act like kernel drivers…

The two main reasons I know/care about (there may be more) are that it can be much faster (zero copy, reduced context switches) and also guarantee the message isn't changed while reading (by sealing the originating buffer). The last is something you need kernel help to enforce.

Yeah, but is it worth the cost? I would say no.

Re: Bus1: a new Linux interprocess communication proposal

#56

Post by [deleted] » Thu, Aug 18, 2016, 11:55 AM UTC

digi_owl wrote: post #38

chme wrote: post #26

Earlier quoted context omitted.

Its just so much easier to add new functionality to the kernel than to remove old ones.

Because then they don't have to deal with Torvalds "we don't break userspace" stance. Especially as the devs involved seems to hate maintaining API compatibility...

[deleted]

Re: Bus1: a new Linux interprocess communication proposal

#57

Post by Annatar » Thu, Aug 18, 2016, 12:05 PM UTC

Nursie wrote: post #44

Annatar wrote: post #42

Earlier quoted context omitted.

to the ability to hire any of a "an army" of past or current kernel developers to help solve my problems or do my custom development Aha, you mean like redhat and IBM did, where there are now "kernel developers" who can't find their way out of a wet paper bag, and spend more time arguing with paying customers back-and-forth in redhat's bug system, because they don't have a clue where the problem is ? When you have so…

You exhibit a weird mixture of snobbery and incompetence. How odd.

Ah, my apologies for that: I did not realize that shoddy NFS software is due to my incompetence. I also apologize that I wasn't able to get something which kind-of worked in RHEL 5 to work in RHEL 6.5 because things which are completely unnecessary on UNIX (like worrying about separate portmapper service) and which were in userland are now suddenly in the kernel. But only for NFS 4. Because the documentation is actually unclear on whether that's the case for NFS 2 and 3. And because the NFS protocol specifies that client and the server should auto-negotiate which version they will use. Because of course the community can't figure out where what should be, so flux. Again.

I guess it's really snobby of me to except things which work on UNIX according to specification to behave the same way on Linux. Because Linux is cool. And it's great. And it has the biggest, bestest community. And because I don't appreciate just how great that is, instead being focused on getting work done with computers.

Now let's rehash inter-process communication again. Because it's fun rehashing things which work in other operating systems, over and over and over in Linux. Because Linux is cool like that.

Re: Bus1: a new Linux interprocess communication proposal

#58

Post by saynsedit » Thu, Aug 18, 2016, 12:42 PM UTC

pjmlp wrote: post #31

chubot wrote: post #13

Earlier quoted context omitted.

I'm very sympathetic to that point of view -- UNIX is a paradox where everybody uses it but most people don't really know how to use it. But this paper makes a good quantitative case for POSIX not meeting the needs of modern computing: https://news.ycombinator.com/item?id=11652609 A good example is that Linux distros, Android, and OS X all have their own non-POSIX IPC mechanisms. Applications are somehow voting with…

> UNIX is a paradox where everybody uses it but most people don't really know how to use it. Specially when the majority nowadays has the idea that GNU/Linux == UNIX and never tried to write an actual portable UNIX application.

Eh? The majority of modern POSIXy software runs on at least Linux, Mac OS X, and FreeBSD.

In fact, Mac OS X is the most widely deployed UNIX-like system ever. What you're a claiming to be true probably more applies to Mac OS X than Linux.

I honestly haven't seen a recent package that was accidentally not portable in the long-term across the various modern UNIX-like systems because of confusion of Linux for POSIX.

Re: Bus1: a new Linux interprocess communication proposal

#59

Post by naasking » Thu, Aug 18, 2016, 12:46 PM UTC

chubot wrote: post #13

nixos wrote: post #3

Can someone please explain what's wrong with the classical UNIX sockets and pipes that they need to invent and (re)invent dbus, bonobo, now bus1?

I'm very sympathetic to that point of view -- UNIX is a paradox where everybody uses it but most people don't really know how to use it. But this paper makes a good quantitative case for POSIX not meeting the needs of modern computing: https://news.ycombinator.com/item?id=11652609 A good example is that Linux distros, Android, and OS X all have their own non-POSIX IPC mechanisms. Applications are somehow voting with…

> It leaves me scratching my head because I don't think serialization is that hard a problem.

It is a hard problem, at least if you want to achieve concision, make it parseable, and support schema upgrade.

> "Traditional POSIX threading models, IPC interfaces, and file system access are being replaced by platform and vendor-specific APIs and frameworks such as Grand Central Dispatch [18], Binder [29], DBus [25], and SQLite [1]."

And good riddance. File systems are almost universally terrible [1]. Threading with shared state is also a terrible idea because it's too general, not to mention the poor APIs used to manage them.

IPC is something that's still unfortunately undergoing constant revision, and the structured binary vs. text wars unfortunately both have merit. I think structured text output via a domain-specific language (DSL) is really the proper answer, somewhat like how a REPL works. This DSL is ultimately simple to construct within C since it would be lisp-like in its structure so it doesn't need an IDL compiler, although its textual syntaz shouldn't be lisp-like.

stdin can then nicely format DSL output for the user. Best of all worlds between text, binary, and structured formats.

[1] https://danluu.com/file-consistency/

Re: Bus1: a new Linux interprocess communication proposal

#60

Post by Annatar » Thu, Aug 18, 2016, 12:46 PM UTC

vidarh wrote: post #28

Annatar wrote: post #16

Earlier quoted context omitted.

That's the thing about Linux: they have literally not one, but multiple armies of "programmers", and yet even after 20+ years of an absolutely furious development pace, basic things like inter-process communication still do not work desirably. Startup/shutdown? Yep, you know it already, it's still in flux (SMF in illumos has been humming along for a decade now). Filesystems? Flux. Observability? Flux. libc, what libc…

It will always be in flux because it reflects what Illumos doesn't have: A userbase as diverse and large as Linux, of which a large proportion have competing needs and wishes and the ability to change it at will . This is not just about license either, but about the availability of people who understand the kernel. I'm sure we could have something cleaner with a single, coherent team. But we also would have something…

Don't underestimate the importance of people.

This warrants a reply of its own, since it is the core idea of my essays: it is the very difference between a stable, working, reliable product and Linux: the people. The right people. Competent people. People with the neccessary education, experience, and insight.

If if ever need heart surgery, I guarantee you that I won't go to the local shaman, no matter how happy he will be to perform the operation voluntarily and for free. Same goes for the local exorcist. If I ever need one, I'll get my surgery done by a specialist in cardiology, because they have been formally educated, formally trained, and have the requisite experience to operate on a human heart. And in the case of illumos / SmartOS, they will even do it gratis, like "medical doctors without borders" organization does. Right people for the job. For some reason, people here and at large believe that when it comes to inter-process communication, NFS, or writing operating system kernels anyone can do it.

Why exactly people believe that computer science is different from surgery, I haven't discovered yet, but based on empirical evidence of Linux and other crappy software, I have a hypothesis, and since the sample size is large, I might have a theory soon...