41–45 of 45 posts
"Try it yourself" "very quickly wanted to get real-time streaming for more" My experience is the opposite. You think you need streaming, so you "try it out" and build something incredibly complex with Kafka, that needs 24h maintenance to monitor congestion in every pipeline. And 10x more expensive because your servers are always up. And some clever (expensive) engineers that figure out how watermarks, out of ordernes…
Kafka integrates against aws lamdas very easily
"Try it yourself" "very quickly wanted to get real-time streaming for more" My experience is the opposite. You think you need streaming, so you "try it out" and build something incredibly complex with Kafka, that needs 24h maintenance to monitor congestion in every pipeline. And 10x more expensive because your servers are always up. And some clever (expensive) engineers that figure out how watermarks, out of ordernes…
A classic "batch job" is one that can be executed without input from a keyboard or output to a display, and therefore can be queues in a batch with other such jobs (perhaps from other programmers).
There is a connection with scripting; batch job control was done with command languages. This is where DOS/Windows "batch files" get their name, and the .BAT suffix.
Grouping transmitted items together (such as bytes into a datagram) is better called aggregation, not to confuse it with "batch job" batching.
Nearly all streaming uses aggregation, other than at the lowest data link and physical layers.