For an example how far the kernel goes to get log messages out even on a dying system and how that's used in real deployments:
https://netflixtechblog.com/kubernetes-and-kernel-panics-ed6...
The real realtime preemption end game
141–150 of 281 posts
Re: The real realtime preemption end game
#142Earlier quoted context omitted.
Does everything runs on those CPUs though? Hard realtime control is often done on much simpler MCU at the lowest level, with oversight/planning for a high level system....
In short, no. For Ingenuity (the Mars2020 helicopter) the flight computer runs on pair of hard-realtime Cortex R5 MCUs paired with a FPGA. The non-realtime Snapdragon SoC handles navigation/image processing duties. https://news.ycombinator.com/item?id=26907669
That's basically what I expected, thanks.
Re: The real realtime preemption end game
#143Synchronous logging strikes again! We ran into this some at work with GLOG (Google's logging library), which can, e.g., block on disk IO if stdout is a file or whatever. GLOG was like, 90-99% of culprits when our service stalled for over 100ms.
[deleted]
Re: The real realtime preemption end game
#144Earlier quoted context omitted.
If you lose logs when your service crashes you're losing logs at the time they are most important.
That's unavoidable if the logging service is down when your server crashes. Having a local queue doesn't mean logging to the service is delayed, it can be sent immediately. All the local queue does is give you some resiliency, by being able to retry if the first logging attempt fails.
If your logging service is down all bets are off. But by buffering logs you're now accepting that problems not related to the logging service will also cause you to drop logs - as I mentioned, your service crashing, or being OOM'd, would be one example.
Re: The real realtime preemption end game
#145Earlier quoted context omitted.
But that's still not guaranteed delivery. You're doing what the OP presented - choosing to drop logs under some circumstances when the system is down. a) If your service crashes and it's in-memory, you lose logs b) If your service can't push logs off (upstream service is down or slow) you either drop logs, run out of memory, or block
You are thinking too much in terms of the stated requirements instead of what people actually want: good uptime and good debugability. Falling back to local logging means a blip in logging availability doesn't turn into all hands on deck everything is on fire. And it means that logs will very likely be available for any failures. In other words it's good enough.
Good enough is literally "best effort delivery", you're just agreeing with them that this is ultimately a distributed systems problem and you either choose CP or AP.
Re: The real realtime preemption end game
#146Re: The real realtime preemption end game
#147QNX had this right decades ago. The microkernel has upper bounds on everything it does. There are only a few tens of thousands of lines of microkernel code. All the microkernel does is allocate memory, dispatch the CPU, and pass messages between processes. Everything else, including drivers and loggers, is in user space and can be preempted by higher priority threads. The QNX kernel doesn't do anything with strings.…
For a modern example, there's seL4. I believe it does no dynamic memory allocation. It's also formally verified for various properties. (Arguably?) its biggest contribution to kernel design is the pervasive usage of capabilities to securely but flexibly export control to userspace.
Capabilities are important, but I don’t think that was introduced by seL4. Mach (which underlies macOS) has the same capability-based system.
Re: The real realtime preemption end game
#148Earlier quoted context omitted.
I get the sense that applications with true realtime requirements generally have hard enough requirements that they cannot allow even the remote possibility of failure. Think avionics, medical devices, automotive, military applications. If you really need realtime, then you really need it and "close enough" doesn't really exist. This is just my perception as an outsider though.
> Think avionics, medical devices, automotive, military applications. FWIW by-device/by-transistor-count, the bulk of "hard realtime systems" with millisecond-scale latency requirements are just audio. The sexy stuff are all real applications too. But mostly we need this just so we don't hear pops and echos in our video calls.
Nobody thinks Teams is a realtime application
Re: The real realtime preemption end game
#149Earlier quoted context omitted.
It may not be safety critical, but remember that people can and will purchase $14k power chords to (ostensibly) improve the experience of listening to "just music". https://www.audioadvice.com/audioquest-nrg-dragon-high-curre...
FWIW, a power chord is a _very_ different thing than a power cord.
LOL, what a typo! Good catch!
Re: The real realtime preemption end game
#150About printk, the backported RT implementation of printk added to the RHEL 9.3 kernel has deadlocks ... https://issues.redhat.com/browse/RHEL-15897 & https://issues.redhat.com/browse/RHEL-9380