Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The real realtime preemption end game

lwn.net

Post reply on HN
Print view View original

91–100 of 281 posts

Re: The real realtime preemption end game

#91

Post by wolverine876 »

msm_ wrote: post #41
cduzz wrote: post #11

Earlier quoted context omitted.

I have discussions with cow-orkers around logging; "We have Best-Effort and Guaranteed-Delivery APIs" "I want Guaranteed Delivery!!!" "If the GD logging interface is offline or slow, you'll take downtime; is that okay?" "NO NO Must not take downtime!" "If you need it logged, and can't log it, what do you do?" These days I just point to the CAP theorem and suggest that logging is the same as any other distributed syst…

Interesting, I'd think logging is one of the clearest situations when you want best effort. Logging is, almost by definition, not the "core" of your application, so failure to log properly should not prevent the core of the program from working. Killing the whole program because logging server is clearly throwing the baby out with the bathwater. What people probably mean is "logging is important, let's avoid losing l…

Logging can be essential to security (to auditing). It's your record of what happened. If an attacker can cause logging to fail, they can cover their tracks more easily.

Re: The real realtime preemption end game

#93

Post by mort96 »

salamanderman wrote: post #88

I had a frustrating number of job interviews in my early career where the interviewers didn't know what realtime actually was. That "and predictable delay" concept from the article frequently seemed to be lost on many folks, who seemed to think realtime just meant fast, whatever that means.

I would even remove the "minimum" part altogether; the point of realtime is that operations have predictable upper bounds. That might even mean slower average cases than in non-realtime systems. If you're controlling a car's braking system, "the average delay is 50ms but might take up to 80ms" might be acceptable, whereas "the average delay is 1ms but it might take arbitrarily long, possibly multiple seconds" isn't.

Re: The real realtime preemption end game

#94

Post by KWxIUElW8Xt0tD9 »

moffkalast wrote: post #87
binary132 wrote: post #13

Earlier quoted context omitted.

I get the sense that applications with true realtime requirements generally have hard enough requirements that they cannot allow even the remote possibility of failure. Think avionics, medical devices, automotive, military applications. If you really need realtime, then you really need it and "close enough" doesn't really exist. This is just my perception as an outsider though.

I feel like at this point we have enough cores (or will soon, anyway) that you could dedicate one entirely to one process and have it run realtime.

That's one way to run DPDK processes under LINUX -- you get the whole processor for doing whatever network processing you want to do -- no interruptions from anything.

Re: The real realtime preemption end game

#95

Post by fnordpiglet »

msm_ wrote: post #41
cduzz wrote: post #11

Earlier quoted context omitted.

I have discussions with cow-orkers around logging; "We have Best-Effort and Guaranteed-Delivery APIs" "I want Guaranteed Delivery!!!" "If the GD logging interface is offline or slow, you'll take downtime; is that okay?" "NO NO Must not take downtime!" "If you need it logged, and can't log it, what do you do?" These days I just point to the CAP theorem and suggest that logging is the same as any other distributed syst…

Interesting, I'd think logging is one of the clearest situations when you want best effort. Logging is, almost by definition, not the "core" of your application, so failure to log properly should not prevent the core of the program from working. Killing the whole program because logging server is clearly throwing the baby out with the bathwater. What people probably mean is "logging is important, let's avoid losing l…

It depends.

Some systems the logs are journaled records for the business or are discoverable artifacts for compliance. In highly secure environments logs are not only durable but measures are taken to fingerprint them and their ordering (like ratchet hashing) to ensure integrity is invariant.

I would note that using disk based logging is generally harmful in these situations IMO. Network based logging is less likely to cause blocking at some OS level or other sorts of jitter that’s harder to mask. Typically I develop logging as an in memory thing that offloads to a remote service over the network. The durability of the memory store can be an issue in highly sensitive workloads, and you’ll want to do synchronous disk IO for that case to ensure durability and consistent time budgets, but for almost all application disk less logging is preferable.

Re: The real realtime preemption end game

#96

Post by mort96 »

hgfghui7 wrote: post #83
insanitybit wrote: post #68

Earlier quoted context omitted.

But that's still not guaranteed delivery. You're doing what the OP presented - choosing to drop logs under some circumstances when the system is down. a) If your service crashes and it's in-memory, you lose logs b) If your service can't push logs off (upstream service is down or slow) you either drop logs, run out of memory, or block

You are thinking too much in terms of the stated requirements instead of what people actually want: good uptime and good debugability. Falling back to local logging means a blip in logging availability doesn't turn into all hands on deck everything is on fire. And it means that logs will very likely be available for any failures. In other words it's good enough.

"Good uptime and good reliability but no guarantees" is just a good best effort system.

Re: The real realtime preemption end game

#97

Post by snickerbockers »

calvinmorrison wrote: post #18
binary132 wrote: post #13

Earlier quoted context omitted.

I get the sense that applications with true realtime requirements generally have hard enough requirements that they cannot allow even the remote possibility of failure. Think avionics, medical devices, automotive, military applications. If you really need realtime, then you really need it and "close enough" doesn't really exist. This is just my perception as an outsider though.

If you really need realtime, and you really actually need it, should you be using a system like Linux at all?

Pretty sure most people who think they need a real-time thread actually don't tbh.

Re: The real realtime preemption end game

#98

Post by NalNezumi »

Slightly tangential, but does anyone know good learning material to understand real-time (Linux) kernel more? For someone with rudimentary Linux knowledge.

I've had to compile&install real-time kernel as a requirement for a robot arm (franka) control computer. It would be nice to know a bit more than just how to install the kernel.

Re: The real realtime preemption end game

#99

Post by ska »

0xDEF wrote: post #58
eisbaw wrote: post #19

Great to hear. However even if Linux the kernel is real-time, likely the hardware won't be due to caches and internal magic CPU trickery. Big complex hardware is a no-no for true real-time. That's why AbsInt and WCET tools mainly has simple CPU architectures. 8051 will truly live forever. btw, Zephyr RTOS.

>Big complex hardware is a no-no for true real-time. SpaceX uses x86 processors for their rockets. That small drone copter NASA put on Mars uses "big-ish" ARM cores that can probably run older versions of Android.

Does everything runs on those CPUs though? Hard realtime control is often done on much simpler MCU at the lowest level, with oversight/planning for a high level system....

Re: The real realtime preemption end game

#100

Post by ajross »

binary132 wrote: post #13
tyingq wrote: post #5

I wonder if this being fixed will result in it displacing some notable amount of made-for-realtime hardware/software combos. Especially since there's now lots of cheap, relatively low power, and high clock rate ARM and x86 chips to choose from. With the clock rates so high, perfect real-time becomes less important as you would often have many cycles to spare for misses. I understand it's less elegant, efficient, etc.…

I get the sense that applications with true realtime requirements generally have hard enough requirements that they cannot allow even the remote possibility of failure. Think avionics, medical devices, automotive, military applications. If you really need realtime, then you really need it and "close enough" doesn't really exist. This is just my perception as an outsider though.

> Think avionics, medical devices, automotive, military applications.

FWIW by-device/by-transistor-count, the bulk of "hard realtime systems" with millisecond-scale latency requirements are just audio.

The sexy stuff are all real applications too. But mostly we need this just so we don't hear pops and echos in our video calls.

Post reply on HN