Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The real realtime preemption end game

lwn.net

Post reply on HN
Print view View original

81–90 of 281 posts

Re: The real realtime preemption end game

#81

Post by tremon »

insanitybit wrote: post #69
msm_ wrote: post #41

Earlier quoted context omitted.

Interesting, I'd think logging is one of the clearest situations when you want best effort. Logging is, almost by definition, not the "core" of your application, so failure to log properly should not prevent the core of the program from working. Killing the whole program because logging server is clearly throwing the baby out with the bathwater. What people probably mean is "logging is important, let's avoid losing l…

If you lose logs when your service crashes you're losing logs at the time they are most important.

But if your service has downtime because the logs could not be written, that seems strictly inferior. As someone else wrote upthread, you only want guaranteed delivery for logs if they're required under a strict audit regime and the cost of noncompliance is higher than the cost of a service outage.

Re: The real realtime preemption end game

#82

Post by salamanderman »

rezonant wrote: post #72
cduzz wrote: post #11

Earlier quoted context omitted.

I have discussions with cow-orkers around logging; "We have Best-Effort and Guaranteed-Delivery APIs" "I want Guaranteed Delivery!!!" "If the GD logging interface is offline or slow, you'll take downtime; is that okay?" "NO NO Must not take downtime!" "If you need it logged, and can't log it, what do you do?" These days I just point to the CAP theorem and suggest that logging is the same as any other distributed syst…

> "If the GD logging interface is offline or slow, you'll take downtime; is that okay?" > [edit: added "GD" to clarify that I was referring to the guaranteed delivery logging api, not the best effort logging API] i read GD as god-damned :-)

me too [EDIT: and I totally empathized]

Re: The real realtime preemption end game

#83

Post by hgfghui7 »

insanitybit wrote: post #68
supriyo-biswas wrote: post #65

Earlier quoted context omitted.

The better way to do this is to write the logs to a file or an in-memory ring buffer and have a separate thread/process push logs from the file/ring-buffer to the logging service, allowing for retries if the logging service is down or slow (for moderately short values of down/slow). Promtail[1] can do this if you're using Loki for logging. [1] https://grafana.com/docs/loki/latest/send-data/promtail/

But that's still not guaranteed delivery. You're doing what the OP presented - choosing to drop logs under some circumstances when the system is down. a) If your service crashes and it's in-memory, you lose logs b) If your service can't push logs off (upstream service is down or slow) you either drop logs, run out of memory, or block

You are thinking too much in terms of the stated requirements instead of what people actually want: good uptime and good debugability. Falling back to local logging means a blip in logging availability doesn't turn into all hands on deck everything is on fire. And it means that logs will very likely be available for any failures.

In other words it's good enough.

Re: The real realtime preemption end game

#84

Post by tremon »

binary132 wrote: post #51
dripton wrote: post #38

Earlier quoted context omitted.

You can divide realtime applications into safety-critical and non-safety-critical ones. For safety-critical apps, you're totally right. For non-critical apps, if it's late and therefore buggy once in a while, that sucks but nobody dies. Examples of the latter include audio and video playback and video games. Nobody wants pauses or glitches, but if you get one once in a while, nobody dies. So people deliver these on n…

This kind of makes the same point I made though -- apps without hard realtime requirements aren't "really realtime" applications

No -- soft realtime applications are things like video conferencing, where you care mostly about low latency in the audio/video stream but it's ok to drop the occasional frame. These are still realtime requirements, different from what your typical browser does (for example): who cares if a webpage is rendered in 100ms or 2s? Hard realtime is more like professional audio/video recording where you want hard guarantees that each captured frame is stored and processed within the alotted time.

Re: The real realtime preemption end game

#85

Post by kbenson »

insanitybit wrote: post #68
supriyo-biswas wrote: post #65

Earlier quoted context omitted.

The better way to do this is to write the logs to a file or an in-memory ring buffer and have a separate thread/process push logs from the file/ring-buffer to the logging service, allowing for retries if the logging service is down or slow (for moderately short values of down/slow). Promtail[1] can do this if you're using Loki for logging. [1] https://grafana.com/docs/loki/latest/send-data/promtail/

But that's still not guaranteed delivery. You're doing what the OP presented - choosing to drop logs under some circumstances when the system is down. a) If your service crashes and it's in-memory, you lose logs b) If your service can't push logs off (upstream service is down or slow) you either drop logs, run out of memory, or block

Yeah, what the "best effort" actually means in practice is usually a result of how much resources you want to throw at the problem. Those give you runway on how much of a problem you can withstand and perhaps recover from without any loss of data (logs), but in the end you're usually still just buying time. That's usually enough though.

Re: The real realtime preemption end game

#86

Post by fnordpiglet »

lopkeny12ko wrote: post #75
loeg wrote: post #4

Synchronous logging strikes again! We ran into this some at work with GLOG (Google's logging library), which can, e.g., block on disk IO if stdout is a file or whatever. GLOG was like, 90-99% of culprits when our service stalled for over 100ms.

I would posit that if your product's availability hinges on +/- 100ms, you are doing something deeply wrong, and it's not your logging library's fault. Users are not going to care if a button press takes 100 more ms to complete.

100ms for something like say API authorization on a high volume data plane service would be unacceptable. Exceeding latencies like that can degrade bandwidth and cause workers to exhaust connection counts. Likewise, even in humans response space, 100ms is an enormous part of a budget for responsiveness. Taking again authorization, if you spend 100ms, you’re exhausting the perceptible threshold for a humans sense of responsiveness to do something that’s of no practical value but is entirely necessary. Your UI developers will be literally camped outside your zoom room with virtual pitch forks night and day.

Re: The real realtime preemption end game

#87

Post by moffkalast »

binary132 wrote: post #13
tyingq wrote: post #5

I wonder if this being fixed will result in it displacing some notable amount of made-for-realtime hardware/software combos. Especially since there's now lots of cheap, relatively low power, and high clock rate ARM and x86 chips to choose from. With the clock rates so high, perfect real-time becomes less important as you would often have many cycles to spare for misses. I understand it's less elegant, efficient, etc.…

I get the sense that applications with true realtime requirements generally have hard enough requirements that they cannot allow even the remote possibility of failure. Think avionics, medical devices, automotive, military applications. If you really need realtime, then you really need it and "close enough" doesn't really exist. This is just my perception as an outsider though.

I feel like at this point we have enough cores (or will soon, anyway) that you could dedicate one entirely to one process and have it run realtime.

Re: The real realtime preemption end game

#88

Post by salamanderman »

I had a frustrating number of job interviews in my early career where the interviewers didn't know what realtime actually was. That "and predictable delay" concept from the article frequently seemed to be lost on many folks, who seemed to think realtime just meant fast, whatever that means.

Re: The real realtime preemption end game

#89

Post by laurencerowe »

sroussey wrote: post #70
supriyo-biswas wrote: post #65

Earlier quoted context omitted.

The better way to do this is to write the logs to a file or an in-memory ring buffer and have a separate thread/process push logs from the file/ring-buffer to the logging service, allowing for retries if the logging service is down or slow (for moderately short values of down/slow). Promtail[1] can do this if you're using Loki for logging. [1] https://grafana.com/docs/loki/latest/send-data/promtail/

We did something like this at Weebly for stats. The app sent the stats to a local service via UDP, so shoot and forget. That service aggregated for 1s and then sent off server.

Why UDP for a local service rather than a unix socket?

Re: The real realtime preemption end game

#90

Post by hamandcheese »

lopkeny12ko wrote: post #75
loeg wrote: post #4

Synchronous logging strikes again! We ran into this some at work with GLOG (Google's logging library), which can, e.g., block on disk IO if stdout is a file or whatever. GLOG was like, 90-99% of culprits when our service stalled for over 100ms.

I would posit that if your product's availability hinges on +/- 100ms, you are doing something deeply wrong, and it's not your logging library's fault. Users are not going to care if a button press takes 100 more ms to complete.

Not every API is a simple CRUD app with a user at the other end.
Post reply on HN