Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gazette: Cloud-native millisecond-latency streaming

github.com

Post reply on HN
Print view View original

21–30 of 53 posts

Re: Gazette: Cloud-native millisecond-latency streaming

#21

Post by CyberDildonics »

modeless wrote: post #19
Groxx wrote: post #11

Earlier quoted context omitted.

The screen in front of your face. That'll give you like 3ns latency.

It is frequently faster to send an IP packet to another continent than to change a pixel on the screen.

No it isn't. John Carmack found a tv ten years ago that had 200-300ms of latency due to all its post processing and wrote an essay about it.

That doesn't mean that it is "frequently" faster to send packets to other continents than change pixels on screens. It doesn't even apply to modern tvs set up for latency, let alone computer monitors.

Re: Gazette: Cloud-native millisecond-latency streaming

#22

Post by amluto »

From reading the docs, this has an IMO surprising design decision: the “journal” is a stream of bytes, where each append (of a byte string) is atomic and occurs in a global order. The bytes are grouped into fragments, and no write spans a fragment boundary.

This seems sort of okay if writes are self-delimiting and never corrupt, and synchronization can always be recovered at a fragment boundary.

I suppose it’s neat that one can write JSONL and get actual JSONL in the blobs. But this seems quite brittle if multiple writers write to one journal and one malfunctions (aside from possibly failing to write a delimiter, there’s no way to tell who wrote a record, and using only a single writer per journal seems to defeat the purpose). And getting, say, Parquet output doesn’t seem like it will happen in any sensible way.

Re: Gazette: Cloud-native millisecond-latency streaming

#23

Post by haberman »

JoshTriplett wrote: post #17
mightyham wrote: post #15

Earlier quoted context omitted.

I don't know anything about the internals of io_uring and am genuinely curious how it works. Saying it "directly works asynchronously" doesn't mean anything though. When circular buffer requests are processed what thread is processing the request, how is that thread managed, and how does it manage blocking/unblocking when communicating with the storage device?

Internally, many parts of the Linux kernel operate asynchronously: they queue up a request with some subsystem (e.g. a hardware device), and get an event delivered when the request is completed. In such cases, io_uring can enqueue such a request, and complete it when receiving the event, without needing to use a thread to block waiting for it. See, for instance, https://lpc.events/event/11/contributions/901/attachmen…

Is it safe to say that a single thread using io_uring should be as fast or faster than N threads performing the same set of I/O tasks in a blocking manner?

In other words, can you count on the kernel to use its own threads internally whenever an I/O task might actually need to use a lot of CPU?

Re: Gazette: Cloud-native millisecond-latency streaming

#24

Post by jauntywundrkind »

amluto wrote: post #20
jauntywundrkind wrote: post #18

Earlier quoted context omitted.

You don't need io_uring. For many workloads being slow & inefficient is acceptable, isn't awful. But gee I'd rather start from a modern baseline that has high levels of mechanistic sympathy with the hardware, where things like network & io work can be done in an efficient async manner. Why do I need io_uring? Because it sounds awful and unhackerly to suffer living in a much lesser worse world.

Mechanical sympathy is understanding the system, not using the shiniest thing. If you want low latency processing of one event at a time, you are either going to burn an entire core spinning or you are going to do a syscall for each operation. The io_uring syscalls are not especially fast — they get their awesomeness by doing, potentially, a whole lot of work per operation. And, for some use cases, by having a superi…

Low latency for a single event is never going to have mechanistic sympathy, will be a colossal waste of most of your system.

Highly concurrent system usage is what it takes. EPOLLEXCLUSIVE (2016) finally sort of gets epoll vaguely capable of what OSes were doing decades ago but is still difficult to use & a rats nest of complexity. Who here feels good reading https://stackoverflow.com/questions/41582560/how-does-epolls... ?

The submission/completion queue model of io_uring makes sense. It lets work be added or resolved without crossing that painful slow kernel barrier. It's been expanded to offer a lot more operations than what could be done in epoll.

The "shiniest thing" is a vast leap in capabilities, systems legibility, and overall (not single operation) throughout. You cannot remotely get the numbers io_uring was bringing three years ago any other way. And it's only gotten further and further ahead while everyone else has sat still.

Re: Gazette: Cloud-native millisecond-latency streaming

#25

Post by modeless »

CyberDildonics wrote: post #21
modeless wrote: post #19

Earlier quoted context omitted.

It is frequently faster to send an IP packet to another continent than to change a pixel on the screen.

No it isn't. John Carmack found a tv ten years ago that had 200-300ms of latency due to all its post processing and wrote an essay about it. That doesn't mean that it is "frequently" faster to send packets to other continents than change pixels on screens. It doesn't even apply to modern tvs set up for latency, let alone computer monitors.

Yes, it really is. The problem is it takes a lot more than one frame for most modern software to change a pixel on the screen. I'm sitting in Hawaii on wifi right now and the first random US mainland server I pinged responded in 120ms, which means sending only took 60ms. Now say you're running a 30 Hz game with 2 frames of input lag, and you've already lost before even considering the input lag of the monitor itself.

There are just so many ways to accidentally get many frames of input lag. OS window compositors generally add a whole frame of input lag globally to every windowed app. Anything running in a browser has a second compositor in between it and the display that can add more frames. GPU APIs typically buffer one or two frames by default. And all of that is on top of whatever the app itself does, and whatever the monitor does (and whatever the input device does if you want to count that too).

Re: Gazette: Cloud-native millisecond-latency streaming

#26

Post by nextaccountic »

modeless wrote: post #25
CyberDildonics wrote: post #21

Earlier quoted context omitted.

No it isn't. John Carmack found a tv ten years ago that had 200-300ms of latency due to all its post processing and wrote an essay about it. That doesn't mean that it is "frequently" faster to send packets to other continents than change pixels on screens. It doesn't even apply to modern tvs set up for latency, let alone computer monitors.

Yes, it really is. The problem is it takes a lot more than one frame for most modern software to change a pixel on the screen. I'm sitting in Hawaii on wifi right now and the first random US mainland server I pinged responded in 120ms, which means sending only took 60ms. Now say you're running a 30 Hz game with 2 frames of input lag, and you've already lost before even considering the input lag of the monitor itself.…

> OS window compositors generally add a whole frame of input lag globally to every windowed app.

Is there a way to verify this is the case? In X11 Linux specifically.

Also does variable refresh rate like freesync help with this?

Re: Gazette: Cloud-native millisecond-latency streaming

#27

Post by Onavo »

danthelion wrote: post #2

Gazette is at the core of Estuary Flow ( https://estuary.dev ), a real-time data platform. Unlike Kafka, Gazette’s architecture is simpler to reason about and operate. It plays well with k8s and is backed by S3 (or any object storage).

Interesting, are there any open source alternatives to tinybird?

https://www.tinybird.co/

Re: Gazette: Cloud-native millisecond-latency streaming

#28

Post by zokier »

jauntywundrkind wrote: post #18
immibis wrote: post #10

Earlier quoted context omitted.

I seem to be missing context for this reply. Why do you need io_uring?

You don't need io_uring. For many workloads being slow & inefficient is acceptable, isn't awful. But gee I'd rather start from a modern baseline that has high levels of mechanistic sympathy with the hardware, where things like network & io work can be done in an efficient async manner. Why do I need io_uring? Because it sounds awful and unhackerly to suffer living in a much lesser worse world.

if you want mechanistic sympathy and low latency then you can't really do much better than dpdk; uring is still going through the very generic and abstracted kernel networking stack.

Re: Gazette: Cloud-native millisecond-latency streaming

#29

Post by 10000truths »

haberman wrote: post #23
JoshTriplett wrote: post #17

Earlier quoted context omitted.

Internally, many parts of the Linux kernel operate asynchronously: they queue up a request with some subsystem (e.g. a hardware device), and get an event delivered when the request is completed. In such cases, io_uring can enqueue such a request, and complete it when receiving the event, without needing to use a thread to block waiting for it. See, for instance, https://lpc.events/event/11/contributions/901/attachmen…

Is it safe to say that a single thread using io_uring should be as fast or faster than N threads performing the same set of I/O tasks in a blocking manner? In other words, can you count on the kernel to use its own threads internally whenever an I/O task might actually need to use a lot of CPU?

If you saturate the submission queue with CPU-bottlenecked tasks, it defeats the value-add of io_uring - at that point, you might as well replace your kernel-space thread pool with a user-space one.

Re: Gazette: Cloud-native millisecond-latency streaming

#30

Post by rswail »

mbrock wrote: post #16
oatmeal_croc wrote: post #13

What's the use case for millisecond-latency streaming? HFT? Remotely driving heavy machinery? Anything else?

I think it's less about guaranteed 1ms real time transactions and more about, like, it's just fast enough that you most likely don't have to worry about it introducing perceptible lag? I'm working on a streaming audio thing and keeping latency low is a priority. I actually think I'll try Gazette, I just saw it now and it was one of those moments where it's like wait I go to Hacker News to waste time but this is quite…

> wait I go to Hacker News to waste time but this is quite exactly what I've been wanting in so many ways.

This has happened so many times for me that I don't consider the time "wasted". I try to make sure I separate the a) "this is interesting personally", and b) "this is interesting professionally" threads and have a bunch of open tabs for a) that I can "read later".

But the items in (b) I read "now" and consider that to be work, not pleasure.

Post reply on HN