Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The real realtime preemption end game

lwn.net

Post reply on HN
Print view View original

11–20 of 281 posts

Re: The real realtime preemption end game

#11

Post by cduzz »

loeg wrote: post #4

Synchronous logging strikes again! We ran into this some at work with GLOG (Google's logging library), which can, e.g., block on disk IO if stdout is a file or whatever. GLOG was like, 90-99% of culprits when our service stalled for over 100ms.

I have discussions with cow-orkers around logging;

"We have Best-Effort and Guaranteed-Delivery APIs"

"I want Guaranteed Delivery!!!"

"If the GD logging interface is offline or slow, you'll take downtime; is that okay?"

"NO NO Must not take downtime!"

"If you need it logged, and can't log it, what do you do?"

These days I just point to the CAP theorem and suggest that logging is the same as any other distributed system. Because there's a wikipedia article with a triangle and the word "theorem" people seem to accept that.

[edit: added "GD" to clarify that I was referring to the guaranteed delivery logging api, not the best effort logging API]

Re: The real realtime preemption end game

#12

Post by imtringued »

tyingq wrote: post #5

I wonder if this being fixed will result in it displacing some notable amount of made-for-realtime hardware/software combos. Especially since there's now lots of cheap, relatively low power, and high clock rate ARM and x86 chips to choose from. With the clock rates so high, perfect real-time becomes less important as you would often have many cycles to spare for misses. I understand it's less elegant, efficient, etc.…

Sure, but this won't magically remove the need for dedicated cores. What will probably happen is that people will tell the scheduler to exclusively put non-premptible real time tasks on one of the LITTLE cores.

Re: The real realtime preemption end game

#13

Post by binary132 »

tyingq wrote: post #5

I wonder if this being fixed will result in it displacing some notable amount of made-for-realtime hardware/software combos. Especially since there's now lots of cheap, relatively low power, and high clock rate ARM and x86 chips to choose from. With the clock rates so high, perfect real-time becomes less important as you would often have many cycles to spare for misses. I understand it's less elegant, efficient, etc.…

I get the sense that applications with true realtime requirements generally have hard enough requirements that they cannot allow even the remote possibility of failure. Think avionics, medical devices, automotive, military applications.

If you really need realtime, then you really need it and "close enough" doesn't really exist.

This is just my perception as an outsider though.

Re: The real realtime preemption end game

#14

Post by nraynaud »

taeric wrote: post #8
EdSchouten wrote: post #6

Earlier quoted context omitted.

Another option is simply to print less, but expose more events in the form of counters. Unfortunately, within a kernel that’s as big as Linux, that would leave you with many, many, many counters. All of which need to be exported and monitored somehow.

This seems to imply you would have more counters than messages? Why would that be? That is, I would expect moving to counters to be less information, period. That not the case?

My guess is that each counter would need to have a discovery point, a regular update mechanism and a documentation, while you can send obscure messages willy-nilly in the log? And also they become an Application Interface with a life cycle while (hopefully) not too many people will go parse the log as an API.

Re: The real realtime preemption end game

#15

Post by tuetuopay »

loeg wrote: post #4

Synchronous logging strikes again! We ran into this some at work with GLOG (Google's logging library), which can, e.g., block on disk IO if stdout is a file or whatever. GLOG was like, 90-99% of culprits when our service stalled for over 100ms.

We had prod halt once when the syslog server hanged. Logs were pushed through TCP which propagated the blocking to the whole of prod. We switched to UDP transport since: better to lose some logs than the whole of prod.

Re: The real realtime preemption end game

#16

Post by eisbaw »

TeeMassive wrote: post #3

It's kind of crazy that a feature necessitated 20 years of active development to be somewhat called complete. I hope it will be ready soon. I'm working in a project that has strict serial communication requirements and it has caused us a lot of headaches.

Zephyr RTOS.

Re: The real realtime preemption end game

#17

Post by tyingq »

tuetuopay wrote: post #10
tyingq wrote: post #5

I wonder if this being fixed will result in it displacing some notable amount of made-for-realtime hardware/software combos. Especially since there's now lots of cheap, relatively low power, and high clock rate ARM and x86 chips to choose from. With the clock rates so high, perfect real-time becomes less important as you would often have many cycles to spare for misses. I understand it's less elegant, efficient, etc.…

The thing is, stuff that require hard realtime cannot satisfy with "many cycles to spare for misses". And CPU cycles is not the whole story. A badly made task could lock down the kernel not doing anything useful. The point of hard realtime is "nothing cannot prevent this critical task from running". For automotive and aerospace, you really want the control systems to be able to run no matter what.

Yes, there are parts of the space that can't be displaced with this.

I'm unclear on why you put "many cycles to spare for misses" in quotes, as if it's unimportant. If a linux/arm (or x86) solution is displacing a much lower speed "real real time" solution, that's the situation...the extra cycles mean you can tolerate some misses while still being as granular as what you're replacing. Not for every use case, but for many.

Re: The real realtime preemption end game

#18

Post by calvinmorrison »

binary132 wrote: post #13
tyingq wrote: post #5

I wonder if this being fixed will result in it displacing some notable amount of made-for-realtime hardware/software combos. Especially since there's now lots of cheap, relatively low power, and high clock rate ARM and x86 chips to choose from. With the clock rates so high, perfect real-time becomes less important as you would often have many cycles to spare for misses. I understand it's less elegant, efficient, etc.…

I get the sense that applications with true realtime requirements generally have hard enough requirements that they cannot allow even the remote possibility of failure. Think avionics, medical devices, automotive, military applications. If you really need realtime, then you really need it and "close enough" doesn't really exist. This is just my perception as an outsider though.

If you really need realtime, and you really actually need it, should you be using a system like Linux at all?

Re: The real realtime preemption end game

#19

Post by eisbaw »

Great to hear. However even if Linux the kernel is real-time, likely the hardware won't be due to caches and internal magic CPU trickery.

Big complex hardware is a no-no for true real-time.

That's why AbsInt and WCET tools mainly has simple CPU architectures. 8051 will truly live forever.

btw, Zephyr RTOS.

Re: The real realtime preemption end game

#20

Post by tetha »

tuetuopay wrote: post #15
loeg wrote: post #4

Synchronous logging strikes again! We ran into this some at work with GLOG (Google's logging library), which can, e.g., block on disk IO if stdout is a file or whatever. GLOG was like, 90-99% of culprits when our service stalled for over 100ms.

We had prod halt once when the syslog server hanged. Logs were pushed through TCP which propagated the blocking to the whole of prod. We switched to UDP transport since: better to lose some logs than the whole of prod.

Especially if some system is unhappy enough to log enough volume to blow up the local log disk... you'll usually have enough messages and clues in the bazillion other messages that have been logged.
Post reply on HN