Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The real realtime preemption end game

lwn.net

Post reply on HN
Print view View original

121–130 of 281 posts

Re: The real realtime preemption end game

#121

Post by SubjectToChange »

eisbaw wrote: post #19

Great to hear. However even if Linux the kernel is real-time, likely the hardware won't be due to caches and internal magic CPU trickery. Big complex hardware is a no-no for true real-time. That's why AbsInt and WCET tools mainly has simple CPU architectures. 8051 will truly live forever. btw, Zephyr RTOS.

Big complex hardware is a no-no for true real-time.

There are advanced real time cores like the Arm Coretex-R82. In fact many real time systems are becoming quite powerful due to the need to process and aggregate ever increasing amounts of sensor data.

Re: The real realtime preemption end game

#122

Post by zokier »

ska wrote: post #99
0xDEF wrote: post #58

Earlier quoted context omitted.

>Big complex hardware is a no-no for true real-time. SpaceX uses x86 processors for their rockets. That small drone copter NASA put on Mars uses "big-ish" ARM cores that can probably run older versions of Android.

Does everything runs on those CPUs though? Hard realtime control is often done on much simpler MCU at the lowest level, with oversight/planning for a high level system....

In short, no. For Ingenuity (the Mars2020 helicopter) the flight computer runs on pair of hard-realtime Cortex R5 MCUs paired with a FPGA. The non-realtime Snapdragon SoC handles navigation/image processing duties.

https://news.ycombinator.com/item?id=26907669

Re: The real realtime preemption end game

#123

Post by Zondartul »

cduzz wrote: post #11
loeg wrote: post #4

Synchronous logging strikes again! We ran into this some at work with GLOG (Google's logging library), which can, e.g., block on disk IO if stdout is a file or whatever. GLOG was like, 90-99% of culprits when our service stalled for over 100ms.

I have discussions with cow-orkers around logging; "We have Best-Effort and Guaranteed-Delivery APIs" "I want Guaranteed Delivery!!!" "If the GD logging interface is offline or slow, you'll take downtime; is that okay?" "NO NO Must not take downtime!" "If you need it logged, and can't log it, what do you do?" These days I just point to the CAP theorem and suggest that logging is the same as any other distributed syst…

I have some wishfull thinking ideas on this, but it should be possible to have both at least in an imaginary, theoretical scenario.

You can have both guaranteed delivery and no downtime if your whole system is so deterministic that anything that normally would result in blocking just will not, cannot happen. In other words it should be a hard real-time system that is formally verified top to bottom, down to the last transistor. Does anyone actually do that? Verify the program and the hardware to prove that it will never run out of memory for logs and such?

Continuing this thought, logs are probably generated endlessly, so either whoever wants them has to also guarantee that that they are processedand disposed of right after being logged... or there is a finite ammount of log messages that can be stored (arbitrary number like 10 000) but the user (of logs) has to guarantee that they will take the "mail" out of the box sooner than it overfills (at some predictable, deterministic rate). So really that means even if OUR system is mathematically perfect, we're just making the downtime someone elses problem - namely, the consumer of the infinite logs.

That, or we guarantee that the final resources of our self-contained, verified system will last longer than the finite shelf life of the system as a whole (like maybe 5 years for another arbitrary number)

Re: The real realtime preemption end game

#124

Post by kccqzy »

lopkeny12ko wrote: post #75
loeg wrote: post #4

Synchronous logging strikes again! We ran into this some at work with GLOG (Google's logging library), which can, e.g., block on disk IO if stdout is a file or whatever. GLOG was like, 90-99% of culprits when our service stalled for over 100ms.

I would posit that if your product's availability hinges on +/- 100ms, you are doing something deeply wrong, and it's not your logging library's fault. Users are not going to care if a button press takes 100 more ms to complete.

Add some fan out and 100ms could suddenly become 1s, 10s…

Re: The real realtime preemption end game

#125

Post by loeg »

fnordpiglet wrote: post #86
lopkeny12ko wrote: post #75

Earlier quoted context omitted.

I would posit that if your product's availability hinges on +/- 100ms, you are doing something deeply wrong, and it's not your logging library's fault. Users are not going to care if a button press takes 100 more ms to complete.

100ms for something like say API authorization on a high volume data plane service would be unacceptable. Exceeding latencies like that can degrade bandwidth and cause workers to exhaust connection counts. Likewise, even in humans response space, 100ms is an enormous part of a budget for responsiveness. Taking again authorization, if you spend 100ms, you’re exhausting the perceptible threshold for a humans sense of r…

Yes, and in fact the service I am talking about is a high volume data plane service.

Re: The real realtime preemption end game

#126

Post by morelisp »

Zondartul wrote: post #123
cduzz wrote: post #11

Earlier quoted context omitted.

I have discussions with cow-orkers around logging; "We have Best-Effort and Guaranteed-Delivery APIs" "I want Guaranteed Delivery!!!" "If the GD logging interface is offline or slow, you'll take downtime; is that okay?" "NO NO Must not take downtime!" "If you need it logged, and can't log it, what do you do?" These days I just point to the CAP theorem and suggest that logging is the same as any other distributed syst…

I have some wishfull thinking ideas on this, but it should be possible to have both at least in an imaginary, theoretical scenario. You can have both guaranteed delivery and no downtime if your whole system is so deterministic that anything that normally would result in blocking just will not, cannot happen. In other words it should be a hard real-time system that is formally verified top to bottom, down to the last…

PACELC says you get blocking or unavailability or inconsistency.

Re: The real realtime preemption end game

#127

Post by nine_k »

tyingq wrote: post #17
tuetuopay wrote: post #10

Earlier quoted context omitted.

The thing is, stuff that require hard realtime cannot satisfy with "many cycles to spare for misses". And CPU cycles is not the whole story. A badly made task could lock down the kernel not doing anything useful. The point of hard realtime is "nothing cannot prevent this critical task from running". For automotive and aerospace, you really want the control systems to be able to run no matter what.

Yes, there are parts of the space that can't be displaced with this. I'm unclear on why you put "many cycles to spare for misses" in quotes, as if it's unimportant. If a linux/arm (or x86) solution is displacing a much lower speed "real real time" solution, that's the situation...the extra cycles mean you can tolerate some misses while still being as granular as what you're replacing. Not for every use case, but for…

How much more expensive and power-hungry an ARM core would be, if it displaces a lower-specced core?

I bet there are hard-realtime (commercial) OSes running on ARM, and the ability to use a lower-specced (cheaper, simpler, consuming less power) core may be seen as an advantage enough to pay for the OS license.

Re: The real realtime preemption end game

#129

Post by cwillu »

itishappy wrote: post #46
cptaj wrote: post #36

Earlier quoted context omitted.

Unless its just music

It may not be safety critical, but remember that people can and will purchase $14k power chords to (ostensibly) improve the experience of listening to "just music". https://www.audioadvice.com/audioquest-nrg-dragon-high-curre...

FWIW, a power chord is a _very_ different thing than a power cord.

Re: The real realtime preemption end game

#130

Post by Animats »

QNX had this right decades ago. The microkernel has upper bounds on everything it does. There are only a few tens of thousands of lines of microkernel code. All the microkernel does is allocate memory, dispatch the CPU, and pass messages between processes. Everything else, including drivers and loggers, is in user space and can be preempted by higher priority threads.

The QNX kernel doesn't do anything with strings. No parsing, no formatting, no messages.

Linux suffers from being too bloated for real time. Millions of lines of kernel, all of which have to be made preemptable. It's the wrong architecture for real time. So it took two decades to try to fix this.

Post reply on HN