Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The real realtime preemption end game

lwn.net

Post reply on HN
Print view View original

61–70 of 281 posts

Re: The real realtime preemption end game

#61

Post by duped »

cptaj wrote: post #36
binary132 wrote: post #13

Earlier quoted context omitted.

I get the sense that applications with true realtime requirements generally have hard enough requirements that they cannot allow even the remote possibility of failure. Think avionics, medical devices, automotive, military applications. If you really need realtime, then you really need it and "close enough" doesn't really exist. This is just my perception as an outsider though.

Unless its just music

Unless that music is being played through a multi kW amplifier into a stadium and an xrun causes damage to the drivers and/or audience (although, they should have hearing protection anyway).

Re: The real realtime preemption end game

#62

Post by fbdab103 »

Aaargh20318 wrote: post #27

What does this mean for the common user? Is this something you would only enable in very specific circumstances or can it also bring a more responsive system to the general public?

My understanding is that real-time makes a system slower. To be real-time, you have to put a time allocation on everything. Each operation is allowed X budget, and will not deviate. This means if the best-case operation is fast, but the worst case is slow, the system has to always assume worst case.

Re: The real realtime preemption end game

#63

Post by 0xDEF »

What do embedded real-time Linux people use for bootloader, init system, utilities, and C standard library implementation? Even Android that does not have real-time constraints ended up using Toybox for utilities and rolling their own C standard library (Bionic).

Re: The real realtime preemption end game

#64

Post by JohnFen »

tyingq wrote: post #5

I wonder if this being fixed will result in it displacing some notable amount of made-for-realtime hardware/software combos. Especially since there's now lots of cheap, relatively low power, and high clock rate ARM and x86 chips to choose from. With the clock rates so high, perfect real-time becomes less important as you would often have many cycles to spare for misses. I understand it's less elegant, efficient, etc.…

When I'm doing realtime applications using cheap, low-power, high-clockrate ARM chips (I don't consider x86 chips for those sorts of applications), I'm not using an operating system at all. An OS interferes too much, even an RTOS. I don't see how this changes anything.

But it all depends on what your application is. There are a lot of applications that are "almost real-time" in need. For those, this might be useful.

Re: The real realtime preemption end game

#65

Post by supriyo-biswas »

cduzz wrote: post #11
loeg wrote: post #4

Synchronous logging strikes again! We ran into this some at work with GLOG (Google's logging library), which can, e.g., block on disk IO if stdout is a file or whatever. GLOG was like, 90-99% of culprits when our service stalled for over 100ms.

I have discussions with cow-orkers around logging; "We have Best-Effort and Guaranteed-Delivery APIs" "I want Guaranteed Delivery!!!" "If the GD logging interface is offline or slow, you'll take downtime; is that okay?" "NO NO Must not take downtime!" "If you need it logged, and can't log it, what do you do?" These days I just point to the CAP theorem and suggest that logging is the same as any other distributed syst…

The better way to do this is to write the logs to a file or an in-memory ring buffer and have a separate thread/process push logs from the file/ring-buffer to the logging service, allowing for retries if the logging service is down or slow (for moderately short values of down/slow).

Promtail[1] can do this if you're using Loki for logging.

[1] https://grafana.com/docs/loki/latest/send-data/promtail/

Re: The real realtime preemption end game

#66

Post by tremon »

tuetuopay wrote: post #50
tyingq wrote: post #17

Earlier quoted context omitted.

Yes, there are parts of the space that can't be displaced with this. I'm unclear on why you put "many cycles to spare for misses" in quotes, as if it's unimportant. If a linux/arm (or x86) solution is displacing a much lower speed "real real time" solution, that's the situation...the extra cycles mean you can tolerate some misses while still being as granular as what you're replacing. Not for every use case, but for…

You won't be saved from two tasks deadlocking with cycles/second. this is what hard realtime systems are about. However, I do agree that not all systems have a real hard realtime requirements. But those usually can handle a non-rt kernel. As for the quotes, it was a direct citation, not a way to dismiss what you said.

I don't think realtime anything has much to do with mutex deadlocks, those are pretty much orthogonal concepts. In fact, I would make a stronger claim: if your "realtime" system can deadlock, it's either not really realtime or it has a design flaw and should be sent back to the drawing board. It's not like you can say "oh, we have a realtime kernel now, so deadlocks are the kernel's problem".

Actual realtime systems are about workload scheduling that takes into account processing deadlines. Hard realtime systems can make guarantees about processing latencies, and can preemptively kill or skip tasks if the result would arrive too late. But this is not something that the Linux kernel can provide, because it is a system property rather than about just the kernel: you can't provide any hard guarantees if you have no time bounds for your data processing workload. So any discussion about -rt in the context of the Linux kernel will always be about soft realtime only.

Re: The real realtime preemption end game

#67

Post by beiller »

duped wrote: post #61
cptaj wrote: post #36

Earlier quoted context omitted.

Unless its just music

Unless that music is being played through a multi kW amplifier into a stadium and an xrun causes damage to the drivers and/or audience (although, they should have hearing protection anyway).

Your talk of xrun is giving me anxiety. When I was younger I dreamed of having a linux audio effects stack with cheap hardware on stage and xruns brought my dreams crashing down.

Re: The real realtime preemption end game

#68

Post by insanitybit »

supriyo-biswas wrote: post #65
cduzz wrote: post #11

Earlier quoted context omitted.

I have discussions with cow-orkers around logging; "We have Best-Effort and Guaranteed-Delivery APIs" "I want Guaranteed Delivery!!!" "If the GD logging interface is offline or slow, you'll take downtime; is that okay?" "NO NO Must not take downtime!" "If you need it logged, and can't log it, what do you do?" These days I just point to the CAP theorem and suggest that logging is the same as any other distributed syst…

The better way to do this is to write the logs to a file or an in-memory ring buffer and have a separate thread/process push logs from the file/ring-buffer to the logging service, allowing for retries if the logging service is down or slow (for moderately short values of down/slow). Promtail[1] can do this if you're using Loki for logging. [1] https://grafana.com/docs/loki/latest/send-data/promtail/

But that's still not guaranteed delivery. You're doing what the OP presented - choosing to drop logs under some circumstances when the system is down.

a) If your service crashes and it's in-memory, you lose logs

b) If your service can't push logs off (upstream service is down or slow) you either drop logs, run out of memory, or block

Re: The real realtime preemption end game

#69

Post by insanitybit »

msm_ wrote: post #41
cduzz wrote: post #11

Earlier quoted context omitted.

I have discussions with cow-orkers around logging; "We have Best-Effort and Guaranteed-Delivery APIs" "I want Guaranteed Delivery!!!" "If the GD logging interface is offline or slow, you'll take downtime; is that okay?" "NO NO Must not take downtime!" "If you need it logged, and can't log it, what do you do?" These days I just point to the CAP theorem and suggest that logging is the same as any other distributed syst…

Interesting, I'd think logging is one of the clearest situations when you want best effort. Logging is, almost by definition, not the "core" of your application, so failure to log properly should not prevent the core of the program from working. Killing the whole program because logging server is clearly throwing the baby out with the bathwater. What people probably mean is "logging is important, let's avoid losing l…

If you lose logs when your service crashes you're losing logs at the time they are most important.

Re: The real realtime preemption end game

#70

Post by sroussey »

supriyo-biswas wrote: post #65
cduzz wrote: post #11

Earlier quoted context omitted.

I have discussions with cow-orkers around logging; "We have Best-Effort and Guaranteed-Delivery APIs" "I want Guaranteed Delivery!!!" "If the GD logging interface is offline or slow, you'll take downtime; is that okay?" "NO NO Must not take downtime!" "If you need it logged, and can't log it, what do you do?" These days I just point to the CAP theorem and suggest that logging is the same as any other distributed syst…

The better way to do this is to write the logs to a file or an in-memory ring buffer and have a separate thread/process push logs from the file/ring-buffer to the logging service, allowing for retries if the logging service is down or slow (for moderately short values of down/slow). Promtail[1] can do this if you're using Loki for logging. [1] https://grafana.com/docs/loki/latest/send-data/promtail/

We did something like this at Weebly for stats. The app sent the stats to a local service via UDP, so shoot and forget. That service aggregated for 1s and then sent off server.
Post reply on HN