Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gazette: Cloud-native millisecond-latency streaming

github.com

Post reply on HN
Print view View original

41–50 of 53 posts

Re: Gazette: Cloud-native millisecond-latency streaming

#41

Post by Groxx »

CyberDildonics wrote: post #35
modeless wrote: post #25

Earlier quoted context omitted.

Yes, it really is. The problem is it takes a lot more than one frame for most modern software to change a pixel on the screen. I'm sitting in Hawaii on wifi right now and the first random US mainland server I pinged responded in 120ms, which means sending only took 60ms. Now say you're running a 30 Hz game with 2 frames of input lag, and you've already lost before even considering the input lag of the monitor itself.…

No it really isn't. Are you really doubling down on this by talking about software that has built in latency? Any game that runs at half the frame rate of a cheap TV and has an architecture designed to not draw frames immediately has nothing to do with what you're saying. That would be like someone deciding to send packets every 100ms and claiming 100ms extra latency. All of this is forgetting that packets can be fir…

Once you throw in head-of-line blocking, other requests in flight, and your average website's pile of ads and JavaScript operating systems layered on top of each other to emulate a small library that reimplements much of what browsers natively support:

Yeah I think displays, even when triple buffered, might win on average. Sending a single packet is fighting a straw man when compared against a full rendering pipeline with common habits. Compare minimums or compare common cases, crossing between them is unfair regardless of which direction you go.

Re: Gazette: Cloud-native millisecond-latency streaming

#42

Post by psfried »

amluto wrote: post #22

From reading the docs, this has an IMO surprising design decision: the “journal” is a stream of bytes , where each append (of a byte string) is atomic and occurs in a global order. The bytes are grouped into fragments, and no write spans a fragment boundary. This seems sort of okay if writes are self-delimiting and never corrupt, and synchronization can always be recovered at a fragment boundary. I suppose it’s neat…

I don't think it's correct to say that JSONL is any more vulnerable to invalid data than other message framings. There's literally no system out there that can fully protect you from bugs in your own application. But the client libraries do validate the framing for you automatically, so in practice the risk is low. I've been running decently large Gazette clusters for years now using the JSONL framing, and have never seen a consumer write invalid JSON to a journal.

The choice of message framing is left to the writers/consumers, so there's also nothing preventing you from using a message framing that you like better. Similarly, there's nothing preventing you from adding metadata that identifies the writer. Having this flexibility can be seen as either a benefit or a pain. If you see it as a pain and want something that's more high-level but less flexible, then you can check out Estuary Flow, which builds on Gazette journals to provide higher-level "Collections" that support many more features.

Re: Gazette: Cloud-native millisecond-latency streaming

#43

Post by amluto »

jgraettinger1 wrote: post #39
amluto wrote: post #22

From reading the docs, this has an IMO surprising design decision: the “journal” is a stream of bytes , where each append (of a byte string) is atomic and occurs in a global order. The bytes are grouped into fragments, and no write spans a fragment boundary. This seems sort of okay if writes are self-delimiting and never corrupt, and synchronization can always be recovered at a fragment boundary. I suppose it’s neat…

:wave: Hi, I'm the creator of Gazette. > But this seems quite brittle if multiple writers write to one journal and one malfunctions (aside from possibly failing to write a delimiter, there’s no way to tell who wrote a record, and using only a single writer per journal seems to defeat the purpose). Yes, writers are responsible for only ever writing complete delimited blocks of messages, in whatever framing the applica…

> :wave: Hi, I'm the creator of Gazette.

Hi!

> if an application correctly writes bad data, then you'll have bad data in your journal. This is no different from any other file format under the sun.

In a journal that delimits itself, a bad write corrupts only that write (and anything depending on it) — it doesn’t make the next message unreadable. I’m not sure how I feel about this.

I maintain a journal-ish thing for internal use, and it’s old and crufty and has all manner of design decisions that, in retrospect, are wrong. But it does strictly separate writes from different sources, and each message has a well defined length.

Also, mine supports compressed files as its source of truth, which is critical for my use case. It looks like Gazette has a way to post process data before it turns into a final fragment — nifty. I wonder whether anyone has rigged it up to produce compressed Parquet files.

Re: Gazette: Cloud-native millisecond-latency streaming

#44

Post by amluto »

jauntywundrkind wrote: post #24
amluto wrote: post #20

Earlier quoted context omitted.

Mechanical sympathy is understanding the system, not using the shiniest thing. If you want low latency processing of one event at a time, you are either going to burn an entire core spinning or you are going to do a syscall for each operation. The io_uring syscalls are not especially fast — they get their awesomeness by doing, potentially, a whole lot of work per operation. And, for some use cases, by having a superi…

Low latency for a single event is never going to have mechanistic sympathy, will be a colossal waste of most of your system. Highly concurrent system usage is what it takes. EPOLLEXCLUSIVE (2016) finally sort of gets epoll vaguely capable of what OSes were doing decades ago but is still difficult to use & a rats nest of complexity. Who here feels good reading https://stackoverflow.com/questions/41582560/how-does-epol…

> Low latency for a single event is never going to have mechanistic sympathy, will be a colossal waste of most of your system.

Excuse me? I maintain a production system that cares about low latency for single events. Declaring that it doesn’t have “mechanistic sympathy” entirely misses the point. Of course I’m not squeezing the most throughput out of every cycle of my CPU. I have a set of design requirements, I understand what the kernel and CPU and IO system do under the hood, and I designed the system to make the most of the resources at hand to achieve the design requirements. Which, in this case, are minimal latency for single events or small groups of events, and io_uring would have no benefit.

(I can steam in events at a very nice rate as measured in events/sec, but I never tried to optimize that, and I should not try to optimize that because it would make the overall system perform worse.)

Re: Gazette: Cloud-native millisecond-latency streaming

#45

Post by 10000truths »

haberman wrote: post #40
10000truths wrote: post #29

Earlier quoted context omitted.

If you saturate the submission queue with CPU-bottlenecked tasks, it defeats the value-add of io_uring - at that point, you might as well replace your kernel-space thread pool with a user-space one.

Sure, but that approach forces you to consider/research just how much CPU your I/O tasks may or may not require. What if I'm not sure? How CPU-intensive is open()? What about close()? What about read()? It would simplify my design process if I could count on io_uring being optimal for ~all I/O tasks, rather than having to treat "CPU-heavy I/O" and "CPU-light I/O" as two separate things that require two separate desig…

This is something that will require profiling to get exact numbers. The non-async portions of a high level filesystem read operation appear rather trivial: checking for cache hits (page cache, dentry cache, etc), parsing the inode/dentry info, and the memcpy to userspace. I wouldn't worry about any of these starving subsequent io_uring SQEs.

I reckon the most likely place you'd find unexpected CPU-heavy work is at the block layer. Software RAID and dmcrypt will burn plenty of cycles, enough to prove as exceptions to the "no FPU instructions in the kernel" guideline.

Re: Gazette: Cloud-native millisecond-latency streaming

#46

Post by hnav »

10000truths wrote: post #5
hnav wrote: post #4

Earlier quoted context omitted.

io_uring is a low level abstraction and is generally a wash against epoll. Really won't make a difference for these kinds of applications, especially not for client nodes.

io_uring allows for async reads and writes to disk without forcing a thread pool or direct I/O. That alone makes it much more scalable for workloads that touch both the network and disk.

The point was that io_uring isn't going to make a big difference for the network code, as for disk I/O code (especially for the sorts of things GP is talking about) you have a bounded number of "threads" of execution anyway. For a node in a pub-sub system, maybe it has c10k users but it's probably appending to a handful of LSM-like datastructures that are written sequentially to disk. The biggest difference is random reads, but even then you can saturate what the disk will do with double digit numbers of threads.

Re: Gazette: Cloud-native millisecond-latency streaming

#47

Post by lossolo »

10000truths wrote: post #45
haberman wrote: post #40

Earlier quoted context omitted.

Sure, but that approach forces you to consider/research just how much CPU your I/O tasks may or may not require. What if I'm not sure? How CPU-intensive is open()? What about close()? What about read()? It would simplify my design process if I could count on io_uring being optimal for ~all I/O tasks, rather than having to treat "CPU-heavy I/O" and "CPU-light I/O" as two separate things that require two separate desig…

This is something that will require profiling to get exact numbers. The non-async portions of a high level filesystem read operation appear rather trivial: checking for cache hits (page cache, dentry cache, etc), parsing the inode/dentry info, and the memcpy to userspace. I wouldn't worry about any of these starving subsequent io_uring SQEs. I reckon the most likely place you'd find unexpected CPU-heavy work is at th…

> Software RAID and dmcrypt will burn plenty of cycles, enough to prove as exceptions to the "no FPU instructions in the kernel" guideline.

LUKS has a negligible impact on I/O bandwidth, and the same is true for software RAID. I'm almost saturating NVMe drives using a combination of LUKS (aes-xts) and software RAID. Additionally, the encryption and decryption processes are almost free when using hardware AES-NI instructions, especially while waiting for I/O.

Re: Gazette: Cloud-native millisecond-latency streaming

#48

Post by immibis »

amluto wrote: post #20
jauntywundrkind wrote: post #18

Earlier quoted context omitted.

You don't need io_uring. For many workloads being slow & inefficient is acceptable, isn't awful. But gee I'd rather start from a modern baseline that has high levels of mechanistic sympathy with the hardware, where things like network & io work can be done in an efficient async manner. Why do I need io_uring? Because it sounds awful and unhackerly to suffer living in a much lesser worse world.

Mechanical sympathy is understanding the system, not using the shiniest thing. If you want low latency processing of one event at a time, you are either going to burn an entire core spinning or you are going to do a syscall for each operation. The io_uring syscalls are not especially fast — they get their awesomeness by doing, potentially, a whole lot of work per operation. And, for some use cases, by having a superi…

Or you are going to use an FPGA network card, like the HFT firms do.

Re: Gazette: Cloud-native millisecond-latency streaming

#49

Post by psfried »

amluto wrote: post #43
jgraettinger1 wrote: post #39

Earlier quoted context omitted.

:wave: Hi, I'm the creator of Gazette. > But this seems quite brittle if multiple writers write to one journal and one malfunctions (aside from possibly failing to write a delimiter, there’s no way to tell who wrote a record, and using only a single writer per journal seems to defeat the purpose). Yes, writers are responsible for only ever writing complete delimited blocks of messages, in whatever framing the applica…

> :wave: Hi, I'm the creator of Gazette. Hi! > if an application correctly writes bad data, then you'll have bad data in your journal. This is no different from any other file format under the sun. In a journal that delimits itself, a bad write corrupts only that write (and anything depending on it) — it doesn’t make the next message unreadable. I’m not sure how I feel about this. I maintain a journal-ish thing for i…

To my knowledge, nobody's implemented parquet fragment files. But it supports compression of JSONL out of the box. JSON compresses very well, and compression ratios approaching 10/1 are not uncommon.

But more to the point, journals are meant for things that are written _and read_ sequentially. Parquet wasn't really designed for sequential reads, so it's unclear to me whether there would be much benefit. IMHO it's better to use journals for sequential data (think change events) and other systems (e.g. RDBMS or parquet + pick-your-compute-flavor) for querying it. I don't think there's yet a storage format that works equally well for both.

Post reply on HN