“Streaming vs. Batch” Is a Wrong Dichotomy, and I Think It's Confusing
1–10 of 45 posts
Re: “Streaming vs. Batch” Is a Wrong Dichotomy, and I Think It's Confusing
#2streaming should be used exclusively for live data...if you keep things that way everything falls smoothly in place
Re: “Streaming vs. Batch” Is a Wrong Dichotomy, and I Think It's Confusing
#3Streams have unknown size and may be infinite.
Batches have a known size and it are not infinite.
Re: “Streaming vs. Batch” Is a Wrong Dichotomy, and I Think It's Confusing
#4streaming should be used exclusively for live data...if you keep things that way everything falls smoothly in place
This gets messy though because the deeper you dig the more the word "live" loses meaning.
Is "live" data something emitted within milliseconds? Seconds? Is it "live" if it’s replayed from a buffer with minimal delay?
Real-time systems aren’t always real-time. Some "live" streams are just batched updates in disguise.
You blink and suddenly you’re in temporal quantum soup.
And once you push the boundaries—high-frequency trading, deep space comms, even global-scale latency—you run into the brick wall of physics. At certain speeds and distances, simultaneity stops being objective. Two observers won’t agree on what "just happened." Causality gets slippery. Streams bifurcate.
At that point, "live" isn’t just fuzzy—it’s frame-dependent. You’re not streaming reality anymore. You’re curating a perspective.
Re: “Streaming vs. Batch” Is a Wrong Dichotomy, and I Think It's Confusing
#5Streams -> optimized for latency
Batches -> optimized for efficiency
Re: “Streaming vs. Batch” Is a Wrong Dichotomy, and I Think It's Confusing
#6If streaming is 5x as expensive as batch, that might be a factor worth considering.
Re: “Streaming vs. Batch” Is a Wrong Dichotomy, and I Think It's Confusing
#7Streams have unknown size and may be infinite. Batches have a known size and it are not infinite.
Maybe I'm using the wrong definitions, but I think that's backwards.
Say you are receiving records from users and different intervals and you want to eventually store them in a different format on a database.
Streaming to me means you're "pushing" to the database according to some rule. For example, wait and accumulate 10 records to push. This could happen in 1 minute or in 10 hours. You know the size of the dataset (exactly 10 records). (You could also add some max time too and then you'd be combining batching with streaming)
Batching to me means you're pulling from the database. For example, you pull once every hour. In that hour, you get 0 records or 1000 records. You don't know the size and it's potentially infinite
Re: “Streaming vs. Batch” Is a Wrong Dichotomy, and I Think It's Confusing
#8It's quite amazing how none of the comments have bothered reading the article, but are also commenting about something completely unrelated to its title.
The article rightfully says that it's not a question of streaming OR batching, because you can stream batches.
Re: “Streaming vs. Batch” Is a Wrong Dichotomy, and I Think It's Confusing
#9Streams have unknown size and may be infinite. Batches have a known size and it are not infinite.
Maybe I'm using the wrong definitions, but I think that's backwards. Say you are receiving records from users and different intervals and you want to eventually store them in a different format on a database. Streaming to me means you're "pushing" to the database according to some rule. For example, wait and accumulate 10 records to push. This could happen in 1 minute or in 10 hours. You know the size of the dataset…
I work with batch oriented store and forward systems and they definitely push data in batches.
Re: “Streaming vs. Batch” Is a Wrong Dichotomy, and I Think It's Confusing
#10It's quite amazing how none of the comments have bothered reading the article, but are also commenting about something completely unrelated to its title. The article rightfully says that it's not a question of streaming OR batching, because you can stream batches.
Because the author is lost in peculiarities of the systems he happens to work with that he is redefining the terms (see all the discussion around "push" and "pull"). That's gonna run into this problem.
His problem is one of data transfer and a better fit for what hes looking for is probably "polling" versus "interrupt" driven.