Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Real time Linux has one known issue remaining

spinics.net

Post reply on HN
Print view View original

11–20 of 26 posts

Re: Real time Linux has one known issue remaining

#11

Post by heavyset_go »

stefan_ wrote: post #8

This headline makes no sense. This particular version has one known issue. Real Time Linux in general of course has a myriad of issues, not least the existence of hypervisors, management controllers and so on in a given piece of hardware that can cause interrupts outside of OS control and therefore unpredictable latency.

Are there processors that don't have management controllers these days? I know ARM processors for phones have things like TrustZone and what not, but what about other ARM processors that are suitable for embedded systems where real-time Linux might make sense?

Re: Real time Linux has one known issue remaining

#12

Post by R0b0t1 »

sudorook wrote: post #10
R0b0t1 wrote: post #7

Earlier quoted context omitted.

Do you have concrete examples of any of those things? The only real tradeoff I am aware of is you sacrifice IO throughput so that the kernel can spread time related service tasks more fairly and userland process can interrupt the kernel. But, as modern Linux kernels work, it is not obvious you experience these throughput problems if you don't mark any tasks with realtime priority. You can still get low power consumpt…

The degree to which the OS deviates from the vanilla kernel behavior depends on the requirements of the processes being run in real-time. For example, to run hard real-time, high frequency (>10 kHz) tasks with low latency and jitter, one needs to disable CPU frequency scaling (among other things) when compiling the kernel. The result is higher power consumption.

You don't have to do those things, it's just easiest to sometimes. It depends entirely on your hardware and your realtime requirements. If you do in-code synchronization you'll usually have problems with throttling but if you can synchronize against a stable timer/counter you will have fewer issues.

It's also worth noting that some of these tasks people throw out are just not a very good fit for even a real time kernel. The main use on Linux seems to be sound and video, which uses specialized hardware. The specialized hardware for other operations is typically not available on Linux capable SoCs and it's worth communicating with a different chip to do those things.

The premise of some of these tradeoffs is not always valid.

Re: Real time Linux has one known issue remaining

#13

Post by EE84M3i »

sudorook wrote: post #10
R0b0t1 wrote: post #7

Earlier quoted context omitted.

Do you have concrete examples of any of those things? The only real tradeoff I am aware of is you sacrifice IO throughput so that the kernel can spread time related service tasks more fairly and userland process can interrupt the kernel. But, as modern Linux kernels work, it is not obvious you experience these throughput problems if you don't mark any tasks with realtime priority. You can still get low power consumpt…

The degree to which the OS deviates from the vanilla kernel behavior depends on the requirements of the processes being run in real-time. For example, to run hard real-time, high frequency (>10 kHz) tasks with low latency and jitter, one needs to disable CPU frequency scaling (among other things) when compiling the kernel. The result is higher power consumption.

Forgive my ignorance, but could you explain a bit more why frequency scaling needs to be disabled at kernel compile time, and can't be disabled at runtime via sysctl?

Re: Real time Linux has one known issue remaining

#14

Post by duped »

capableweb wrote: post #6
etaioinshrdlu wrote: post #2

Will this eventually get merged into the main Linux branch? Are there any downsides to using the real time kernel for more mainstream tasks?

> Are there any downsides to using the real time kernel for more mainstream tasks? Real-time kernels tend to sacrifice anything in order to minimize latency, for being better for use cases where lowest latency is needed. This usually means tradeoffs regarding throughput, power consumption, resource usage, reliability and more, in order to get the lowest possible latency. In some cases you need really low latency, but…

Slight nit, real time systems are designed for a maximum worst case latency within some spec, not minimum latency. You can make a system that has a lower best case latency but that doesn't matter if the worst case causes a missed deadline.

And in the context of real time systems, a missed deadline is either equivalent to a catastrophic error (hard real time) or really terrible but non catastrophic error (soft real time). Soft real time applications like audio (no one dies because of a buffer underrun) can sacrifice the maximum worst case for better average case, because the worst case depends on load and can be worked around. Hard real time cares a lot more about the maximum worst case, like say a CNC machine where missing a deadline can cause a mechanical failure and break the part.

Re: Real time Linux has one known issue remaining

#15

Post by marcan_42 »

etaioinshrdlu wrote: post #2

Will this eventually get merged into the main Linux branch? Are there any downsides to using the real time kernel for more mainstream tasks?

It's already being merged; PREEMPT_RT got merged last year. We're close to the endgame on that front.

There are some performance tradeoffs to using an RT kernel; you trade some throughput for determinism. That said, I've been running RT on my desktop most of the time for years and never had any issues.

Re: Real time Linux has one known issue remaining

#16

Post by marcan_42 »

heavyset_go wrote: post #11
stefan_ wrote: post #8

This headline makes no sense. This particular version has one known issue. Real Time Linux in general of course has a myriad of issues, not least the existence of hypervisors, management controllers and so on in a given piece of hardware that can cause interrupts outside of OS control and therefore unpredictable latency.

Are there processors that don't have management controllers these days? I know ARM processors for phones have things like TrustZone and what not, but what about other ARM processors that are suitable for embedded systems where real-time Linux might make sense?

Apple Silicon machines have no TrustZone, SMM, or anything else to steal control away from the main CPU cores. They also have tricks like being able to direct low-priority IRQs away from CPUs running RT tasks, which we might want to figure out how to support on Linux. The CPU frequency switching latencies are also lower than x86 machines as far as I can tell, and the stalls on switching shorter too. I am very much looking forward to testing out RT on them and seeing how stable I can get low latency audio, compared to typical Intel boxes which are... not great.

Re: Real time Linux has one known issue remaining

#17

Post by pabs3 »

marcan_42 wrote: post #15
etaioinshrdlu wrote: post #2

Will this eventually get merged into the main Linux branch? Are there any downsides to using the real time kernel for more mainstream tasks?

It's already being merged; PREEMPT_RT got merged last year. We're close to the endgame on that front. There are some performance tradeoffs to using an RT kernel; you trade some throughput for determinism. That said, I've been running RT on my desktop most of the time for years and never had any issues.

After all the patches are merged, will there be work on making PREEMPT_RT a boot-time or run-time choice rather than build-time?

Why did you choose RT for your desktop?

Re: Real time Linux has one known issue remaining

#18

Post by sudorook »

EE84M3i wrote: post #13
sudorook wrote: post #10

Earlier quoted context omitted.

The degree to which the OS deviates from the vanilla kernel behavior depends on the requirements of the processes being run in real-time. For example, to run hard real-time, high frequency (>10 kHz) tasks with low latency and jitter, one needs to disable CPU frequency scaling (among other things) when compiling the kernel. The result is higher power consumption.

Forgive my ignorance, but could you explain a bit more why frequency scaling needs to be disabled at kernel compile time, and can't be disabled at runtime via sysctl?

I elided a lot detail for the sake of making a simple example about power consumption.

For applications that don't need hard real-time and/or aren't very high frequency, one could set the CPU frequency governor to always run at max frequency. You're right in that this can be done at runtime without a need to disable CONFIG_CPU_FREQ.

To achieve hard real-time at high frequency (roughly >10kHz), the standard kernel and PREEMPT_RT are not sufficient. One approach is to use a microkernel that preempts the Linux kernel. RTAI and Xenomai are two frameworks that do this. Processes run through them cannot be preempted by the kernel, so deterministic timings can be much better guaranteed. It's in this case that one would disable CPU scaling (along with Intel pstates, ACPI power management, and other things I don't remember offhand) when compiling the kernel. Changes in CPU frequency can change the frequency of the time counters on the CPU, and that opens up a vector for the vanilla kernel to interfere with the timers used by the microkernel and potentially break real-time.

Now, is it true in this latter scenario that one, as I said, "needs to" disable CPU scaling entirely? Now that you've got me thinking about it, probably not. If one guarantees that the 'performance' (i.e. max freq) governor is running at boot, leaving CONFIG_CPU_FREQ enabled may be fine. (The default governor will still need to be changed to the performance one in the kernel config, though.) Setting max frequency at runtime, however, maybe not. This is probably an edge case (I haven't tested this), but if the system boots with a dynamic CPU frequency enabled, the standard kernel, which cannot access information about threads controlled by the microkernel, could scale down the CPU a real-time thread is running on and mess up the timings. Changing the frequency mode at runtime after that happens may not fix that issue.

If I were to write that again, I'd use "should" or instead of "needs to." It's much simpler to disable frequency scaling in the menuconfig while disabling everything else than it is to leave it in and worry about it afterward.

None of this changes the point about real-time and power consumption, though.

Re: Real time Linux has one known issue remaining

#19

Post by op00to »

pabs3 wrote: post #17
marcan_42 wrote: post #15

Earlier quoted context omitted.

It's already being merged; PREEMPT_RT got merged last year. We're close to the endgame on that front. There are some performance tradeoffs to using an RT kernel; you trade some throughput for determinism. That said, I've been running RT on my desktop most of the time for years and never had any issues.

After all the patches are merged, will there be work on making PREEMPT_RT a boot-time or run-time choice rather than build-time? Why did you choose RT for your desktop?

Id consider RT if I was serious about audio.

Re: Real time Linux has one known issue remaining

#20

Post by marcan_42 »

pabs3 wrote: post #17
marcan_42 wrote: post #15

Earlier quoted context omitted.

It's already being merged; PREEMPT_RT got merged last year. We're close to the endgame on that front. There are some performance tradeoffs to using an RT kernel; you trade some throughput for determinism. That said, I've been running RT on my desktop most of the time for years and never had any issues.

After all the patches are merged, will there be work on making PREEMPT_RT a boot-time or run-time choice rather than build-time? Why did you choose RT for your desktop?

I do audio stuff :)
Post reply on HN