Live data from Hacker News

The real realtime preemption end game

lwn.net

171–180 of 281 posts

Re: The real realtime preemption end game

#171
post #67
post #61

Earlier quoted context omitted.

Unless that music is being played through a multi kW amplifier into a stadium and an xrun causes damage to the drivers and/or audience (although, they should have hearing protection anyway).

Your talk of xrun is giving me anxiety. When I was younger I dreamed of having a linux audio effects stack with cheap hardware on stage and xruns brought my dreams crashing down.

xrun definition: https://unix.stackexchange.com/questions/199498/what-are-xru...

(I didn't know the term, trying to be helpful if others don't)

Re: The real realtime preemption end game

#172
post #127
post #17

Earlier quoted context omitted.

Yes, there are parts of the space that can't be displaced with this. I'm unclear on why you put "many cycles to spare for misses" in quotes, as if it's unimportant. If a linux/arm (or x86) solution is displacing a much lower speed "real real time" solution, that's the situation...the extra cycles mean you can tolerate some misses while still being as granular as what you're replacing. Not for every use case, but for…

How much more expensive and power-hungry an ARM core would be, if it displaces a lower-specced core? I bet there are hard-realtime (commercial) OSes running on ARM, and the ability to use a lower-specced (cheaper, simpler, consuming less power) core may be seen as an advantage enough to pay for the OS license.

> How much more expensive and power-hungry an ARM core would be, if it displaces a lower-specced core?

The power issue is real, but it might well be the same price or cheaper - a standard ARM that gets stamped out by the million can cost less than a "simpler" microcontroller with a smaller production run.

Re: The real realtime preemption end game

#173

QNX had this right decades ago. The microkernel has upper bounds on everything it does. There are only a few tens of thousands of lines of microkernel code. All the microkernel does is allocate memory, dispatch the CPU, and pass messages between processes. Everything else, including drivers and loggers, is in user space and can be preempted by higher priority threads. The QNX kernel doesn't do anything with strings.…

QNX is used in vehicle infotainment systems no? Where else? I'm not bothered by the kernel bloat. There's a lot of dev time being invested in Linux and while the desktop is not as much of a priority as say the server space a performant kernel on handhelds and other such devices and the dev work to get it there will benefit the desktop users like myself.

[deleted]

Re: The real realtime preemption end game

#174

QNX had this right decades ago. The microkernel has upper bounds on everything it does. There are only a few tens of thousands of lines of microkernel code. All the microkernel does is allocate memory, dispatch the CPU, and pass messages between processes. Everything else, including drivers and loggers, is in user space and can be preempted by higher priority threads. The QNX kernel doesn't do anything with strings.…

QNX is used in vehicle infotainment systems no? Where else? I'm not bothered by the kernel bloat. There's a lot of dev time being invested in Linux and while the desktop is not as much of a priority as say the server space a performant kernel on handhelds and other such devices and the dev work to get it there will benefit the desktop users like myself.

It was used in my old toyota avensis from 2012. The infotainment was so slow you could measure performance in seconds pr frame instead of frames pr second :)

In the end, all I could practically use it for was as a bluetooth audio connector.

Re: The real realtime preemption end game

#175
post #155
post #22

Earlier quoted context omitted.

I'm guessing it's not that technical experts will be choosing this path, but rather companies. Once it's "good enough", and much easier to hire for, etc...you hire non-experts because it works most of the time. I'm not saying it's good, just that it's a somewhat likely outcome. And not for everything, just the places where they can get away with it.

nah. when functional safety enters the room (as it does for hard real time) then engineers go to jail if they sign off something unsafe and people die because of that. since the challenger disaster there is an awareness that not listening to engineers can be expensive and cost lifes.

[deleted]

Re: The real realtime preemption end game

#176
post #5

I wonder if this being fixed will result in it displacing some notable amount of made-for-realtime hardware/software combos. Especially since there's now lots of cheap, relatively low power, and high clock rate ARM and x86 chips to choose from. With the clock rates so high, perfect real-time becomes less important as you would often have many cycles to spare for misses. I understand it's less elegant, efficient, etc.…

I get the sense that applications with true realtime requirements generally have hard enough requirements that they cannot allow even the remote possibility of failure. Think avionics, medical devices, automotive, military applications. If you really need realtime, then you really need it and "close enough" doesn't really exist. This is just my perception as an outsider though.

Like many binary distinctions, when you zoom in on the details hard-versus-soft realtime is really more of a spectrum. There's "people will die if it's late". "The line will have to stop for a day if it's late". "If it's late, it'll wreck the part currently being made". Etc.

Even hard-realtime systems have a failure rate, in practice if not in theory - even a formally verified system might encounter a hardware bug. So it's always a case of tradeoffs between failure rate and other factors (like cost). If commodity operating systems can push their failure rate down a few orders of magnitude, that moves the needle, at least for some applications.

Re: The real realtime preemption end game

#177
post #101

Earlier quoted context omitted.

It depends. Some systems the logs are journaled records for the business or are discoverable artifacts for compliance. In highly secure environments logs are not only durable but measures are taken to fingerprint them and their ordering (like ratchet hashing) to ensure integrity is invariant. I would note that using disk based logging is generally harmful in these situations IMO. Network based logging is less likely…

If you're not waiting for the remote log server to write the messages to its disk before proceeding, then it seems like that's not guaranteed to me? And if you are, then you suffer all the problems of local disk logging but also all the extra failure modes introduced by the network, too

> If you're not waiting for the remote log server to write the messages to its disk before proceeding, then it seems like that's not guaranteed to me?

Depends on your failure model. I'd consider e.g. "received in memory by at least 3/5 remote servers in separate datacenters" to be safer than "committed to local disk".

Re: The real realtime preemption end game

#179
post #2

What do other realtime OS kernels do when printing from various places? It almost seems like this should be done in hardware because it's such a difficult problem to not lose messages but also have them on a different OS thread in most cases.

In many problem spaces you can optimize for the common success and failure paths if you accept certain losses on long-tail failure scenarios.

A common logging strategy is to use a ring buffer with a separate isolated process reading from the ring. The vast majority of the time the ring buffer handles temporary disruptions (eg slow disk I/O to write messages to disk) but in the rare failure scenarios you simply overwrite events in the buffer and increment an atomic overwritten event counter. Events do not get silently dropped but you prioritize forward progress at the cost of data loss in rare scenarios.

Microkernels and pushing everything to userspace just moves the tradeoffs around. If your driver is in userspace and blocks writing a log message because the log daemon is blocked or the I/O device it is writing the log to is overloaded it does the same thing. Your realtime thread won't get what it needs from the driver within your time limit.

It all comes down to CAP theorem stuff. If you always want the kernel (or any other software) to be able to make forward progress within specific time limits then you must be willing to tolerate some data loss in failure scenarios. How much and how often it happens depends on specific design factors, memory usage, etc.

Re: The real realtime preemption end game

#180

Earlier quoted context omitted.

> QNX had this right decades ago. The microkernel has upper bounds on everything it does. There are only a few tens of thousands of lines of microkernel code. All the microkernel does is allocate memory, dispatch the CPU, and pass messages between processes. Everything else, including drivers and loggers, is in user space and can be preempted by higher priority threads. So much like a well structured main method in a…

Recently, I've been thinking that we need a microkernel design in applications. You have the core and then services that can integrate amongst each other and the core that provide flexibility. Like the "browser as an OS" kind of things but applied more generally.

Yes! This reminds me strongly of the core/modules architecture of the apache httpd, as described by the excellent O'Reilly book on it.

The process of serving an HTTP request is broken into a large number of fine grained stages and plugin modules may hook into any or all of these to modify the input and output to each stage.

The same basic idea makes it easy to turn any application concept into a modules-and-core architecture. From the day I read (skimmed) that book a decade or two ago this pattern has been burned into my brain

Post reply on HN