Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The real realtime preemption end game

lwn.net

Post reply on HN
Print view View original

111–120 of 281 posts

Re: The real realtime preemption end game

#111

Post by [deleted] »

rezonant wrote: post #72
cduzz wrote: post #11

Earlier quoted context omitted.

I have discussions with cow-orkers around logging; "We have Best-Effort and Guaranteed-Delivery APIs" "I want Guaranteed Delivery!!!" "If the GD logging interface is offline or slow, you'll take downtime; is that okay?" "NO NO Must not take downtime!" "If you need it logged, and can't log it, what do you do?" These days I just point to the CAP theorem and suggest that logging is the same as any other distributed syst…

> "If the GD logging interface is offline or slow, you'll take downtime; is that okay?" > [edit: added "GD" to clarify that I was referring to the guaranteed delivery logging api, not the best effort logging API] i read GD as god-damned :-)

[deleted]

Re: The real realtime preemption end game

#112

Post by ReactiveJelly »

fnordpiglet wrote: post #95
msm_ wrote: post #41

Earlier quoted context omitted.

Interesting, I'd think logging is one of the clearest situations when you want best effort. Logging is, almost by definition, not the "core" of your application, so failure to log properly should not prevent the core of the program from working. Killing the whole program because logging server is clearly throwing the baby out with the bathwater. What people probably mean is "logging is important, let's avoid losing l…

It depends. Some systems the logs are journaled records for the business or are discoverable artifacts for compliance. In highly secure environments logs are not only durable but measures are taken to fingerprint them and their ordering (like ratchet hashing) to ensure integrity is invariant. I would note that using disk based logging is generally harmful in these situations IMO. Network based logging is less likely…

If it's a journaled record for the business then I think I'd write it to SQLite or something with good transactions and not mix it in the debug logs

Re: The real realtime preemption end game

#113

Post by oneepic »

loeg wrote: post #4

Synchronous logging strikes again! We ran into this some at work with GLOG (Google's logging library), which can, e.g., block on disk IO if stdout is a file or whatever. GLOG was like, 90-99% of culprits when our service stalled for over 100ms.

Oh, we had this type of issue ("logging lib breaks everything") with a $MSFT logging library. Imagine having 100 threads each with their own logging buffer of 300MB. Needless to say it annihilated our memory and our server crashed, even on the most expensive sku of Azure App Service.

Re: The real realtime preemption end game

#114

Post by RobertVDB »

loeg wrote: post #4

Synchronous logging strikes again! We ran into this some at work with GLOG (Google's logging library), which can, e.g., block on disk IO if stdout is a file or whatever. GLOG was like, 90-99% of culprits when our service stalled for over 100ms.

Ah, the classic GLOG-induced service stall - brings back memories! I've seen similar scenarios where logging, meant to be a safety net, turns into a trap. Your 90-99% figure resonates with my experience. It's like opening a small window for fresh air and having a storm barrel in. We eventually had to balance between logging verbosity and system performance, kind of like a tightrope walk over a sea of unpredictable IO delays. Makes one appreciate the delicate art of designing logging systems that don't end up hogging the spotlight (and resources) themselves, doesn't it?

Re: The real realtime preemption end game

#115

Post by dooglius »

wholesomepotato wrote: post #24
eisbaw wrote: post #19

Great to hear. However even if Linux the kernel is real-time, likely the hardware won't be due to caches and internal magic CPU trickery. Big complex hardware is a no-no for true real-time. That's why AbsInt and WCET tools mainly has simple CPU architectures. 8051 will truly live forever. btw, Zephyr RTOS.

Features of modern CPUs don't really prevent them from real time usage, afaik. As long as something is bounded and can be reasoned about it can be used to build a real time system. You can always assume no cache hits and alikes, maximum load etc and as long as you can put a bound on the time it will take, you're good to go.

System management mode is one example of a feature on modern CPUs that prevents real-time usage https://wiki.linuxfoundation.org/realtime/documentation/howt...

Re: The real realtime preemption end game

#116

Post by refulgentis »

lumb63 wrote: post #26
refulgentis wrote: post #21

Earlier quoted context omitted.

...yes, after realtime support lands

A lot of realtime systems don’t have sufficient resources to run Linux. Their hardware is much less powerful than Linux requires. Even if a system can run (RT-)Linux, it doesn’t mean it’s suitable for real-time. Hardware for real-time projects needs much lower interrupt latency than a lot of hardware provides. Preemption isn’t the only thing necessary to support real-time requirements.

Sure but that was already mentioned before the comment I was replying to. Standard hardware not being great for realtime has nothing to do with hypothetical realtime Linux.

Re: The real realtime preemption end game

#117

Post by ajross »

vlovich123 wrote: post #109
ajross wrote: post #9

Earlier quoted context omitted.

It's just hard, and there's no single answer. In Zephyr, we have a synchronous printk() too, as for low-level debugging and platform bringup that's usually desirable (i.e. I'd like to see the dump from just before the panic please!). For production logging use, though, there is a fancier log system[1] designed around latency boundaries that essentially logs a minimally processed stream to a buffer than then gets flus…

What if the lower priority thread is starved and the buffer is full? Do you start dropping messages? Or overwrite the oldest ones and skip messages?

It drops messages. That's almost always the desired behavior: you never want your logging system to be doing work when the system is productively tasked with other things.

I know there was some level of argument about whether it's best to overwrite older content (ring-buffer-style, probably keeps the most important stuff) or drop messages at input time (faster, probably fewer messages dropped overall). But logging isn't my area of expertise and I forget the details.

But again, the general point being that this is a complicated problem with tradeoffs, where most developers up the stack tend to think of it as a fixed facility that shouldn't ever fail or require developer bandwidth. And it's not, it's hard.

Re: The real realtime preemption end game

#118

Post by [deleted] »

cduzz wrote: post #11
loeg wrote: post #4

Synchronous logging strikes again! We ran into this some at work with GLOG (Google's logging library), which can, e.g., block on disk IO if stdout is a file or whatever. GLOG was like, 90-99% of culprits when our service stalled for over 100ms.

I have discussions with cow-orkers around logging; "We have Best-Effort and Guaranteed-Delivery APIs" "I want Guaranteed Delivery!!!" "If the GD logging interface is offline or slow, you'll take downtime; is that okay?" "NO NO Must not take downtime!" "If you need it logged, and can't log it, what do you do?" These days I just point to the CAP theorem and suggest that logging is the same as any other distributed syst…

[deleted]

Re: The real realtime preemption end game

#120

Post by tuetuopay »

deathanatos wrote: post #107
tuetuopay wrote: post #15

Earlier quoted context omitted.

We had prod halt once when the syslog server hanged. Logs were pushed through TCP which propagated the blocking to the whole of prod. We switched to UDP transport since: better to lose some logs than the whole of prod.

TCP vs. UDP and async best-effort vs. synchronous are completely orthogonal… E.g., a service I wrote wrote logs to an ELK setup; we logged over TCP. But the logging was async: we didn't wait for logs to make it to ELK, and if the logging services went down, we just queued up logs locally. (To a point; at some point, the buffer fills up, and logs were discarded. The process would make a note of this if it happened, lo…

> TCP vs. UDP and async best-effort vs. synchronous are completely orthogonal…

I agree, when stuff is properly written. I don't remember the exact details, but at least with UDP the asyncness is built-in: there is no backpressure whatsoever. So poorly written software can just send udp to heart's end.

Post reply on HN