Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#1

Post by benjamindavy » Wed, Apr 18, 2018, 3:14 PM UTC

Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery
medium.com

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#2

Post by lima » Wed, Apr 18, 2018, 3:31 PM UTC

I built a similar pipeline using Kafka and ClickHouse - it's amazing how easy it is nowadays to ingest and analyze billions of events a day using standard tools.

ClickHouse can even ingest directly from Kafka (courtesy of Cloudflare - http://github.com/vavrusa contributed it).

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#3

Post by xstartup » Wed, Apr 18, 2018, 3:41 PM UTC

Here is an example Pipeline which supports loading data from an unbounded source to BigQuery in batches using load jobs (evading BigQuery's Streaming Insert cost)

See: https://zero-master.github.io/posts/pub-sub-bigquery-beam/

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#4

Post by SiempreViernes » Wed, Apr 18, 2018, 3:43 PM UTC

lima wrote: post #2

I built a similar pipeline using Kafka and ClickHouse - it's amazing how easy it is nowadays to ingest and analyze billions of events a day using standard tools. ClickHouse can even ingest directly from Kafka (courtesy of Cloudflare - http://github.com/vavrusa contributed it).

Do you learn anything useful about humanity tho?

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#5

Post by ryanworl » Wed, Apr 18, 2018, 3:48 PM UTC

For anyone who wants to move this much data out of AWS and into another cloud provider, Kinesis Streams does not charge for bandwidth out. If you do the math, it works out to a very large savings over doing public internet data transfers if you are moving terabytes per day. The compute costs to copy data from Kafka to Kinesis should be minimal since you’re essentially just operating a pipe and not doing much actual compute, and this can operate easily on spot instances.

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#6

Post by pcarolan » Wed, Apr 18, 2018, 4:00 PM UTC

SiempreViernes wrote: post #4

lima wrote: post #2

I built a similar pipeline using Kafka and ClickHouse - it's amazing how easy it is nowadays to ingest and analyze billions of events a day using standard tools. ClickHouse can even ingest directly from Kafka (courtesy of Cloudflare - http://github.com/vavrusa contributed it).

Do you learn anything useful about humanity tho?

Fair comment. The title would imply there's a 'so what?' in the article, not just a 'how'.

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#7

Post by ishi » Wed, Apr 18, 2018, 4:19 PM UTC

lima wrote: post #2

I built a similar pipeline using Kafka and ClickHouse - it's amazing how easy it is nowadays to ingest and analyze billions of events a day using standard tools. ClickHouse can even ingest directly from Kafka (courtesy of Cloudflare - http://github.com/vavrusa contributed it).

Can you elaborate a bit about the servers you used (hardware, cluster size, etc.)?

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#8

Post by davidbrent » Wed, Apr 18, 2018, 4:21 PM UTC

Very interesting read, but as more of an analyst, I kept waiting for the ‘meaning.’ Maybe I missed it, but for anyone else who may be considering reading this, it is more about ‘giving business structure’ to analytic events.

Good article none the less.

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#9

Post by cobookman » Wed, Apr 18, 2018, 4:27 PM UTC

xstartup wrote: post #3

Here is an example Pipeline which supports loading data from an unbounded source to BigQuery in batches using load jobs (evading BigQuery's Streaming Insert cost) See: https://zero-master.github.io/posts/pub-sub-bigquery-beam/

FYI dataflow's bigqueryio does not use stream inserts. Instead it batches the data into many load jobs

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#10

Post by hinkley » Wed, Apr 18, 2018, 4:29 PM UTC

I see titles like this and my first thought is “more people need to watch Real Genius or watch it again.”

“What do you think a phase conjugate tracking system is for, Kent?”

Great. You made a system to track a billion people a day. You’re murdering privacy and then bragging about it. And bragging about it during a giant shitstorm caused by Facebook. The fuck is wrong with you?