Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The real realtime preemption end game

lwn.net

Post reply on HN
Print view View original

131–140 of 281 posts

Re: The real realtime preemption end game

#131

Post by rcxdude »

calvinmorrison wrote: post #18
binary132 wrote: post #13

Earlier quoted context omitted.

I get the sense that applications with true realtime requirements generally have hard enough requirements that they cannot allow even the remote possibility of failure. Think avionics, medical devices, automotive, military applications. If you really need realtime, then you really need it and "close enough" doesn't really exist. This is just my perception as an outsider though.

If you really need realtime, and you really actually need it, should you be using a system like Linux at all?

really depends on your paranoia level and the consequences for failure. soft to hard realtime is a bit of a spectrum in terms of how hard of a failure missing a deadline actually is, and therefore how much you try to verify that you will actually meet that deadline.

Re: The real realtime preemption end game

#132

Post by rcxdude »

lumb63 wrote: post #26
refulgentis wrote: post #21

Earlier quoted context omitted.

...yes, after realtime support lands

A lot of realtime systems don’t have sufficient resources to run Linux. Their hardware is much less powerful than Linux requires. Even if a system can run (RT-)Linux, it doesn’t mean it’s suitable for real-time. Hardware for real-time projects needs much lower interrupt latency than a lot of hardware provides. Preemption isn’t the only thing necessary to support real-time requirements.

realtime just means execution time is bounded. It doesn't necessarily mean the latency is low. Though, in this sense RT-linux should probably be mostly thought of as low-latency linux, and the improvement in realtime guarantees is mostly in reducing the amount of things that can cause you to miss a deadline as opposed to allowing you to guarantee any particular deadline, even a long one.

Re: The real realtime preemption end game

#133

Post by o11c »

insanitybit wrote: post #68
supriyo-biswas wrote: post #65

Earlier quoted context omitted.

The better way to do this is to write the logs to a file or an in-memory ring buffer and have a separate thread/process push logs from the file/ring-buffer to the logging service, allowing for retries if the logging service is down or slow (for moderately short values of down/slow). Promtail[1] can do this if you're using Loki for logging. [1] https://grafana.com/docs/loki/latest/send-data/promtail/

But that's still not guaranteed delivery. You're doing what the OP presented - choosing to drop logs under some circumstances when the system is down. a) If your service crashes and it's in-memory, you lose logs b) If your service can't push logs off (upstream service is down or slow) you either drop logs, run out of memory, or block

Logging to `mmap`ed files is resilient to service crashes, just not hardware crashes.

Re: The real realtime preemption end game

#134

Post by rcxdude »

Aaargh20318 wrote: post #27

What does this mean for the common user? Is this something you would only enable in very specific circumstances or can it also bring a more responsive system to the general public?

the most common desktop end-user that might benefit from this is those doing audio work: latency and especially jitter can be quite a pain there.

Re: The real realtime preemption end game

#135

Post by fnordpiglet »

shawnz wrote: post #101
fnordpiglet wrote: post #95

Earlier quoted context omitted.

It depends. Some systems the logs are journaled records for the business or are discoverable artifacts for compliance. In highly secure environments logs are not only durable but measures are taken to fingerprint them and their ordering (like ratchet hashing) to ensure integrity is invariant. I would note that using disk based logging is generally harmful in these situations IMO. Network based logging is less likely…

If you're not waiting for the remote log server to write the messages to its disk before proceeding, then it seems like that's not guaranteed to me? And if you are, then you suffer all the problems of local disk logging but also all the extra failure modes introduced by the network, too

The difference is that network IO can be more easily masked by the operating system than block device IO. When you offload your logging to another thread the story isn’t over because your disk logging can interfere at a system level. Network IO isn’t as noisy. If durability is important you might still need to wait for an ACK before freeing the buffer for the message which might lead to more overall memory use, all the operations play nicely in a preemptable scheduling system.

Also, the failure modes of systems are very tied to durable storage devices attached to the system and very rarely to network devices. By reducing the number of things that need a disk (ideally to zero) you can remove disks from the system and its availability story. Once you get to fully disk less systems the system failure modes are actually almost nothing. But even with disks attached reducing the times you interact with the disk (especially for chatty things like logs!) reduces the likelihood the entire system fails due to a disk issue.

Re: The real realtime preemption end game

#136

Post by fnordpiglet »

ReactiveJelly wrote: post #112
fnordpiglet wrote: post #95

Earlier quoted context omitted.

It depends. Some systems the logs are journaled records for the business or are discoverable artifacts for compliance. In highly secure environments logs are not only durable but measures are taken to fingerprint them and their ordering (like ratchet hashing) to ensure integrity is invariant. I would note that using disk based logging is generally harmful in these situations IMO. Network based logging is less likely…

If it's a journaled record for the business then I think I'd write it to SQLite or something with good transactions and not mix it in the debug logs

There are more logs than debug logs, and using SQLite as the encoding store for your logs doesn’t make it not logging.

Re: The real realtime preemption end game

#137

Post by rcxdude »

0xDEF wrote: post #63

What do embedded real-time Linux people use for bootloader, init system, utilities, and C standard library implementation? Even Android that does not have real-time constraints ended up using Toybox for utilities and rolling their own C standard library (Bionic).

You aren't likely to need to change a lot of these: the whole point is basically making it so that all that can run as normal but won't really get in the way of your high-priority process. It's just that your high-priority process needs to be careful not to block on anything that might take too long due to some other stuff running. In which case you may need to avoid certain C standard library calls, but not replace it entirely.

Re: The real realtime preemption end game

#138

Post by vacuity »

Animats wrote: post #130

QNX had this right decades ago. The microkernel has upper bounds on everything it does. There are only a few tens of thousands of lines of microkernel code. All the microkernel does is allocate memory, dispatch the CPU, and pass messages between processes. Everything else, including drivers and loggers, is in user space and can be preempted by higher priority threads. The QNX kernel doesn't do anything with strings.…

For a modern example, there's seL4. I believe it does no dynamic memory allocation. It's also formally verified for various properties. (Arguably?) its biggest contribution to kernel design is the pervasive usage of capabilities to securely but flexibly export control to userspace.

Re: The real realtime preemption end game

#140

Post by pests »

oneepic wrote: post #113
loeg wrote: post #4

Synchronous logging strikes again! We ran into this some at work with GLOG (Google's logging library), which can, e.g., block on disk IO if stdout is a file or whatever. GLOG was like, 90-99% of culprits when our service stalled for over 100ms.

Oh, we had this type of issue ("logging lib breaks everything") with a $MSFT logging library. Imagine having 100 threads each with their own logging buffer of 300MB. Needless to say it annihilated our memory and our server crashed, even on the most expensive sku of Azure App Service.

Brilliant strategy.

Reminds me a litte of the oldtimers trick of adding a sleep(1000) somewhere so they could later come back and have some resources later, or if they needed a quick win with the client.

Now cloud companies are using malloc(300000000) it to fake resource usage. /s

Post reply on HN