Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The real realtime preemption end game

lwn.net

Post reply on HN
Print view View original

221–230 of 281 posts

Re: The real realtime preemption end game

#221

Post by dataflow »

rsaxvc wrote: post #219
dataflow wrote: post #199

I feel like focusing on the kernel side misses CPU level issues. Is there any known upper bound on, say, how long a memory access instruction takes on x86?

I don't know for x86. But for things that really matter, I've tested by configuring the MMU to disable caching for the memory that the realtime code lives in and uses to emulate 0% hitrate. And there's usually still a fair amount of variance on top of that depending on if the memory controller has a small cache, and where the memory controller is in its refresh cycle.

Yeah. And I'm not sure that even that would give you the worst case as far as the cache is concerned. Of course I don't know how these implementations work, but it seems plausible that code that directly uses memory could run faster than code that encounters a cache miss beforehand (or contention, if you're using multiple cores). Moreover there's also the instruction cache, and I'm not sure if you can disable caching for that in a meaningful way?

For soft real time, I don't see a problem. But for hard real time, it seems a bit scary.

Re: The real realtime preemption end game

#222

Post by lmm »

cduzz wrote: post #200
lmm wrote: post #177

Earlier quoted context omitted.

> If you're not waiting for the remote log server to write the messages to its disk before proceeding, then it seems like that's not guaranteed to me? Depends on your failure model. I'd consider e.g. "received in memory by at least 3/5 remote servers in separate datacenters" to be safer than "committed to local disk".

You're still on one side or another of the CAP triangle. In a network partition, you are either offline or your data is not consistent. If you're writing local to your system, you're losing data if there's a single device failure. https://en.wikipedia.org/wiki/CAP_theorem

CAP is irrelevant, consistency does not matter for logs.

Re: The real realtime preemption end game

#223

Post by saagarjha »

lopkeny12ko wrote: post #75
loeg wrote: post #4

Synchronous logging strikes again! We ran into this some at work with GLOG (Google's logging library), which can, e.g., block on disk IO if stdout is a file or whatever. GLOG was like, 90-99% of culprits when our service stalled for over 100ms.

I would posit that if your product's availability hinges on +/- 100ms, you are doing something deeply wrong, and it's not your logging library's fault. Users are not going to care if a button press takes 100 more ms to complete.

Core libraries at, say, Google, are supposed to be reliable to several nines. If they go down for long enough for a human to notice, they’re failing SLA.

Re: The real realtime preemption end game

#224

Post by saagarjha »

dataflow wrote: post #199

I feel like focusing on the kernel side misses CPU level issues. Is there any known upper bound on, say, how long a memory access instruction takes on x86?

You can continually take page faults in a Turing complete way without executing any code, so I would guess this is unbounded?

Re: The real realtime preemption end game

#225

Post by HankB99 »

rsaxvc wrote: post #215
HankB99 wrote: post #203

Earlier quoted context omitted.

Interesting to mention the Raspberry Pi. I saw an article just a day or two ago that claimed that the RpiOS was stated by and ran on top of RTOS. That's particularly interesting because at one time years ago, I saw suggestions that Linux could run as a task on an RTOS. Things that required hard real time deadlines could run on the RTOS and not be subject to the delays that a virtual memory system could entail. I don'…

>That's particularly interesting because at one time years ago, I saw suggestions that Linux could run as a task on an RTOS. I've worked with systems that ran Linux as a task of uITRON as well as threadX, both on somewhat obscure ARM hardware. Linux managed the MMU but had a large carveout for the RTOS code. They had some strange interrupt management so that Linux could 'disable interrupts' but while Linux IRQs were…

Interesting to know that it was more than just an idea - thanks!

Re: The real realtime preemption end game

#226

Post by dataflow »

saagarjha wrote: post #224
dataflow wrote: post #199

I feel like focusing on the kernel side misses CPU level issues. Is there any known upper bound on, say, how long a memory access instruction takes on x86?

You can continually take page faults in a Turing complete way without executing any code, so I would guess this is unbounded?

I almost mentioned page faults, but that's something the kernel has control over. It could just make sure everything is in memory so there aren't any faults. So it's not really an issue I think.

Re: The real realtime preemption end game

#227

Post by vbezhenar »

vacuity wrote: post #170
gigatexal wrote: post #162

Earlier quoted context omitted.

> QNX had this right decades ago. The microkernel has upper bounds on everything it does. There are only a few tens of thousands of lines of microkernel code. All the microkernel does is allocate memory, dispatch the CPU, and pass messages between processes. Everything else, including drivers and loggers, is in user space and can be preempted by higher priority threads. So much like a well structured main method in a…

Recently, I've been thinking that we need a microkernel design in applications. You have the core and then services that can integrate amongst each other and the core that provide flexibility. Like the "browser as an OS" kind of things but applied more generally.

COM, OSGI, Service architecture, microservice architecture and countless other approaches. This is correct way to build applications, because it gets reinvented over and over again.

Re: The real realtime preemption end game

#228

Post by adastra22 »

mananaysiempre wrote: post #218
vacuity wrote: post #204

Earlier quoted context omitted.

Yes, but I believe seL4 took it to the max. I may be wrong on that count, but I think seL4 is unique in that it leverages capabilities for pretty much everything except the scheduler. (There was work in that area, but it's incomplete.)

IIRC the KeyKOS/EROS/CapROS tradition used capabilities for everything including the scheduler. Of course, pervasive persistence makes those systems somewhat esoteric (barring fresh builds, they never shut down or boot up, only go to sleep and wake up in new bodies; compare Smalltalk, etc.).

Amoeba was my favorite, as it was a homogeneous, decentralized operating system. Different CPU architectures spread across different data centers, and it was all homogenized together into a single system image. You had a shell prompt where you typed commands and the OS could decide to spawn your process on your local device, in the server room rack, or in some connected datacenter in Amsterdam, it didn't make a difference. From the perspective of you, your program, or the shell, it's just a giant many-core machine with weird memory and peripheral access latencies that the OS manages.

Oh, and anytime as needed the OS could serialize out your process, pipe it across the network to another machine, and resume. Useful for load balancing, or relocating a program to be near the data it is accessing. Unless your program pays special attention to the clock, it wouldn't notice.

I still think about Amoeba from time to time, and imagine what could have been if we had gone down that route instead.

Re: The real realtime preemption end game

#229

Post by zmgsabst »

LeifCarrotson wrote: post #110
zmgsabst wrote: post #47

Earlier quoted context omitted.

What’s an example of a system that requires hard real time and couldn’t cope with soft real time on a 3GHz system having 1000 cycle misses costing 0.3us?

We've successfully used a Delta Tau real-time Linux motion controller to run a 24 kHz laser galvo system. It's ostensibly good for 25 microsecond loop rates, and pretty intolerant of jitter (you could delay a measurement by a full loop period if you're early). And the processor is a fixed frequency Arm industrial deal that only runs at 1.2 GHz. Perhaps even that's not an example of such a system, 0.3 microseconds is…

Thanks for the detailed reply!

I’m trying to understand where the roadblock on a rPi + small FPGA hybrid board for $50 fails at the task… and it sounds like the OS/firmware doesn’t suffice. (Or a SoC, like a Zynq.)

Eg, if we could guarantee that the 1.5GHz core won’t “be off” by more than 1us on responding and the FPGA can manage IO directly to buffer out (some of) the jitter, then the cost of many hobby systems with “(still not quite) hard” real time systems would come down to reasonable.

Re: The real realtime preemption end game

#230

Post by worthless-trash »

TeeMassive wrote: post #33
worthless-trash wrote: post #23

Earlier quoted context omitted.

Can you expand on this, as I'm a little naive in this area, say you isolated the cpus (isolcpus parameter) and then taskset your task onto the isolated cpu, would not the scheduler no longer be involved, and your task be the only thing serviced by that CPU ? Is it other interrupts on the CPU that break your process out of the "real time" requirement, I find this all so interesting.

It's an embedded system with two logical cores with at least 4 other critical processes running. Doing that will only displace the problem.

I (incorrectly) assumed that serial port control was the highly sensitive time problem that was being dealt with here.
Post reply on HN