With the Spark direct stream, kafka partitions are 1:1 with spark partitions, which means at most half of the workers would be doing work without a shuffle.

Seems like a pretty basic oversight that should be addressed.

Re: Benchmarking Streaming Computation Engines at Yahoo

Post by luckydata » Sat, Dec 19, 2015, 5:18 AM UTC

estefan wrote: post #2

This is the first mention I've seen of flink on HN.

there were a few before but not many.

Post reply on HN