The real realtime preemption end game
1–10 of 281 posts
Re: The real realtime preemption end game
#2What do other realtime OS kernels do when printing from various places? It almost seems like this should be done in hardware because it's such a difficult problem to not lose messages but also have them on a different OS thread in most cases.
Re: The real realtime preemption end game
#3It's kind of crazy that a feature necessitated 20 years of active development to be somewhat called complete.
I hope it will be ready soon. I'm working in a project that has strict serial communication requirements and it has caused us a lot of headaches.
Re: The real realtime preemption end game
#4Synchronous logging strikes again! We ran into this some at work with GLOG (Google's logging library), which can, e.g., block on disk IO if stdout is a file or whatever. GLOG was like, 90-99% of culprits when our service stalled for over 100ms.
Re: The real realtime preemption end game
#5I wonder if this being fixed will result in it displacing some notable amount of made-for-realtime hardware/software combos. Especially since there's now lots of cheap, relatively low power, and high clock rate ARM and x86 chips to choose from. With the clock rates so high, perfect real-time becomes less important as you would often have many cycles to spare for misses.
I understand it's less elegant, efficient, etc. But sometimes commodity wins over correctness.
Re: The real realtime preemption end game
#6What do other realtime OS kernels do when printing from various places? It almost seems like this should be done in hardware because it's such a difficult problem to not lose messages but also have them on a different OS thread in most cases.
Another option is simply to print less, but expose more events in the form of counters.
Unfortunately, within a kernel that’s as big as Linux, that would leave you with many, many, many counters. All of which need to be exported and monitored somehow.
Re: The real realtime preemption end game
#7I wonder if this being fixed will result in it displacing some notable amount of made-for-realtime hardware/software combos. Especially since there's now lots of cheap, relatively low power, and high clock rate ARM and x86 chips to choose from. With the clock rates so high, perfect real-time becomes less important as you would often have many cycles to spare for misses. I understand it's less elegant, efficient, etc.…
Ethernet nods in vigorous agreement
Re: The real realtime preemption end game
#8What do other realtime OS kernels do when printing from various places? It almost seems like this should be done in hardware because it's such a difficult problem to not lose messages but also have them on a different OS thread in most cases.
Another option is simply to print less, but expose more events in the form of counters. Unfortunately, within a kernel that’s as big as Linux, that would leave you with many, many, many counters. All of which need to be exported and monitored somehow.
This seems to imply you would have more counters than messages? Why would that be?
That is, I would expect moving to counters to be less information, period. That not the case?
Re: The real realtime preemption end game
#9What do other realtime OS kernels do when printing from various places? It almost seems like this should be done in hardware because it's such a difficult problem to not lose messages but also have them on a different OS thread in most cases.
It's just hard, and there's no single answer.
In Zephyr, we have a synchronous printk() too, as for low-level debugging and platform bringup that's usually desirable (i.e. I'd like to see the dump from just before the panic please!).
For production logging use, though, there is a fancier log system[1] designed around latency boundaries that essentially logs a minimally processed stream to a buffer than then gets flushed from a low priority thread. And this works, and avoids the kinds of problems detailed in the linked article. But it's fiddly to configure, expensive in an RTOS environment (you need RAM for that thread stack and the buffer), depends on having a I/O backend that is itself async/low-latency, and has the mentioned misfeature where when things blow up, it's usually failed to flush the information you need out of its buffer.
[1] Somewhat but not completely orthogonal with printk. Both can be implemented in terms of each others, mostly. Sometimes.
Re: The real realtime preemption end game
#10I wonder if this being fixed will result in it displacing some notable amount of made-for-realtime hardware/software combos. Especially since there's now lots of cheap, relatively low power, and high clock rate ARM and x86 chips to choose from. With the clock rates so high, perfect real-time becomes less important as you would often have many cycles to spare for misses. I understand it's less elegant, efficient, etc.…
The thing is, stuff that require hard realtime cannot satisfy with "many cycles to spare for misses". And CPU cycles is not the whole story. A badly made task could lock down the kernel not doing anything useful. The point of hard realtime is "nothing cannot prevent this critical task from running".
For automotive and aerospace, you really want the control systems to be able to run no matter what.