31–40 of 45 posts
"Try it yourself" "very quickly wanted to get real-time streaming for more" My experience is the opposite. You think you need streaming, so you "try it out" and build something incredibly complex with Kafka, that needs 24h maintenance to monitor congestion in every pipeline. And 10x more expensive because your servers are always up. And some clever (expensive) engineers that figure out how watermarks, out of ordernes…
Kafka integrates against aws lamdas very easily
Earlier quoted context omitted.
The lines blur though when you start keeping state between batches, and a lot of batch processing ends up requiring that (joins, deduplication, etc).
No, it really doesn't. The definition of "streaming", to me, can be boiled down to "you send individual data as soon as it's available, without collecting into groups." Batching is, by definition, the gathering of data records into a collection before you send it. Streaming does not do that, which is the entire point. What happens after transmission occurs, on reception, is entirely irrelevant to whether the data tra…
Earlier quoted context omitted.
No, it really doesn't. The definition of "streaming", to me, can be boiled down to "you send individual data as soon as it's available, without collecting into groups." Batching is, by definition, the gathering of data records into a collection before you send it. Streaming does not do that, which is the entire point. What happens after transmission occurs, on reception, is entirely irrelevant to whether the data tra…
Most streaming does some batching. If you stream audio from a live source, you batch at least into "frames", and you batch into network packets. On top of that you might batch further depending on your requirements, yet I would still count most of it as "streaming".
An audio stream can easily be defined as a series of records, where each record is a sample spanning N seconds, probably as provided by the hardware. Similarly, a video frame can also be considered a record. As soon as a record becomes available, it is sent. Thus, streaming.
Optimizing to fully utilize network frames can generally be considered a low level transport optimization, and thus not relevant to the discussion.
Earlier quoted context omitted.
The lines blur though when you start keeping state between batches, and a lot of batch processing ends up requiring that (joins, deduplication, etc).
No, it really doesn't. The definition of "streaming", to me, can be boiled down to "you send individual data as soon as it's available, without collecting into groups." Batching is, by definition, the gathering of data records into a collection before you send it. Streaming does not do that, which is the entire point. What happens after transmission occurs, on reception, is entirely irrelevant to whether the data tra…
"Try it yourself" "very quickly wanted to get real-time streaming for more" My experience is the opposite. You think you need streaming, so you "try it out" and build something incredibly complex with Kafka, that needs 24h maintenance to monitor congestion in every pipeline. And 10x more expensive because your servers are always up. And some clever (expensive) engineers that figure out how watermarks, out of ordernes…
Kafka integrates against aws lamdas very easily
You can only really take one garden gnome, put some paint on it and forward the same gnome.
Earlier quoted context omitted.
Because the author is lost in peculiarities of the systems he happens to work with that he is redefining the terms (see all the discussion around "push" and "pull"). That's gonna run into this problem. His problem is one of data transfer and a better fit for what hes looking for is probably "polling" versus "interrupt" driven.
Interrupts are a hardware feature on CPUs. You could have software that is effectively checking for events on each tick (clock cycle), and emulates interrupts. But that's what polling is.
Earlier quoted context omitted.
Kafka integrates against aws lamdas very easily
You can't do groupbys or joins with lambdas. You can only really take one garden gnome, put some paint on it and forward the same gnome.
Earlier quoted context omitted.
You can't do groupbys or joins with lambdas. You can only really take one garden gnome, put some paint on it and forward the same gnome.
I think I get the analogy, something like: you can append to the record but everything is still record-based? And what do garden gnomes have to do with it :-)