Sounds like they over-engineered the solution. If you have ad-hoc use-case, BigQuery is great but it's quite expensive. If you just need to pre-calculate the metrics using SQL, Athena / Prestodb / Clickhouse / Redshift Spectrum might be much easier and cost-efficient.
Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery
11–20 of 70 posts
Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery
#12I see titles like this and my first thought is “more people need to watch Real Genius or watch it again.” “What do you think a phase conjugate tracking system is for, Kent?” Great. You made a system to track a billion people a day. You’re murdering privacy and then bragging about it. And bragging about it during a giant shitstorm caused by Facebook. The fuck is wrong with you?
I want to know how people use my website, fuck me right?
Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery
#13Here is an example Pipeline which supports loading data from an unbounded source to BigQuery in batches using load jobs (evading BigQuery's Streaming Insert cost) See: https://zero-master.github.io/posts/pub-sub-bigquery-beam/
Are you the author? OT but I'm amazed the author charged merely 100 euro for implementing that solution for the subject startup, even if they're cash-strapped. I'm not familiar with BigQuery, but I'm curious what a normal rate for solving that issue would look like.
Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery
#14Sounds like they over-engineered the solution. If you have ad-hoc use-case, BigQuery is great but it's quite expensive. If you just need to pre-calculate the metrics using SQL, Athena / Prestodb / Clickhouse / Redshift Spectrum might be much easier and cost-efficient.
I agree with this statement if you consider BigQuery's on-demand pricing, but at a given scale the $40,000/month flat rate becomes competitive.
Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery
#15I see titles like this and my first thought is “more people need to watch Real Genius or watch it again.” “What do you think a phase conjugate tracking system is for, Kent?” Great. You made a system to track a billion people a day. You’re murdering privacy and then bragging about it. And bragging about it during a giant shitstorm caused by Facebook. The fuck is wrong with you?
I want to know how people use my website, fuck me right?
Exactly, fuck you.
Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery
#16Sounds like they over-engineered the solution. If you have ad-hoc use-case, BigQuery is great but it's quite expensive. If you just need to pre-calculate the metrics using SQL, Athena / Prestodb / Clickhouse / Redshift Spectrum might be much easier and cost-efficient.
BigQuery PM here. I'd love to genuinely understand why you have that impression.
BigQuery's on-demand model charges you EXACTLY for what you consume. Meaning, your resource efficiency is 100% [0].
By contrast, typical "cluster pricing" technologies require you to pay for 100% of your cluster uptime. In private data centers, it's difficult to get above 30% average efficiency.
BigQuery also takes care of all software, security, and hardware maintenance, including reprocessing data in our storage system for maximum performance and scaling your BigQuery "cluster" for you.[1]
BigQuery has a perpetual free tier of 10GB of data stored and 1TB of data processed per month.
Finally, BigQuery is the only technology we're aware of whose logical storage system doesn't charge you for loads - meaning we don't compromise your query capacity, nor do we bill you for loads.
[0] https://cloud.google.com/blog/big-data/2016/02/visualizing-t...
[1] https://cloud.google.com/blog/big-data/2016/08/google-bigque...
Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery
#17I see titles like this and my first thought is “more people need to watch Real Genius or watch it again.” “What do you think a phase conjugate tracking system is for, Kent?” Great. You made a system to track a billion people a day. You’re murdering privacy and then bragging about it. And bragging about it during a giant shitstorm caused by Facebook. The fuck is wrong with you?
I want to know how people use my website, fuck me right?
As soon as the IP packets leave your server, it’s not “your website” anymore.
Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery
#18Here is an example Pipeline which supports loading data from an unbounded source to BigQuery in batches using load jobs (evading BigQuery's Streaming Insert cost) See: https://zero-master.github.io/posts/pub-sub-bigquery-beam/
FYI dataflow's bigqueryio does not use stream inserts. Instead it batches the data into many load jobs
That is configurable, but for unbounded PCollections it appears the default is to use streaming inserts https://beam.apache.org/documentation/sdks/javadoc/2.4.0/org...
Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery
#19For anyone who wants to move this much data out of AWS and into another cloud provider, Kinesis Streams does not charge for bandwidth out. If you do the math, it works out to a very large savings over doing public internet data transfers if you are moving terabytes per day. The compute costs to copy data from Kafka to Kinesis should be minimal since you’re essentially just operating a pipe and not doing much actual c…
Can you elaborate on that? I haven't been able to conclude that from what it says on the pricing page [1]:
> Data transfer is free. AWS does not charge for data transfer from your data producers to Amazon Kinesis Data Streams, or from Amazon Kinesis Data Streams to your Amazon Kinesis Applications.
That sounds like it's sonly free to Amazon Kinesis Applications (== inside AWS).
And on [2] it says:
> If you use Amazon EC2 for running your Amazon Kinesis Applications, you will be charged for Amazon EC2 resources in addition to Amazon Kinesis Data Streams costs.
So that sounds like you will eventually pay the normal egress cost of EC2.
[1]: https://aws.amazon.com/kinesis/data-streams/pricing/ [2]: https://aws.amazon.com/kinesis/data-streams/faqs/
Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery
#20You 'd be surprised how good modern hardware is. We are being hit with between 1.5 and 2 million requests per minute (steady traffic no spikes except increased usage in the weekends), and our analytics solution runs on just 2 main servers and costs $3k per month (total associated costs except human labour which is 1 engineer working on it part time now).
We talked with a famous analytics company and they gave us a quote of 1 million yearly to work with us. So how did we get it down from 1m to 35k?
We pretty much do the thing the article suggests (roll-ups section). We compute data hourly. Alongside with the hourly computation we also dump some extra data that can be used to compute numbers for the day (eg unique user ids, from unique user ids per hour), then from the day we get per week and then per month. We also have less moving pieces (our stack is way more traditional), and we manage our own hardware (key for keeping costs down)
When you get data back from the system you only hit the pre-computed cache, no query touches the main system from the dashboards. We only allow queries running in a 30 minute window to run on the live system - to ensure that no crazy load is going to be built on top of it and we use that to mostly catch anomalies on the real time data. (our parsing time is good too, between 10 seconds and 1 minute compared to the 2-30 minutes the article gives).
However this is the "You are alive but you 're not living" angst of analytics. All the data is there, but you cannot freely prod it for answers and patterns. If you want to get an answers about past data, you need to go through an overly complex process of raising a new cluster and ingesting old backups, multiple times, then waiting for a few days. It get's relatively expensive, slow and at times will demoralize you and make you back off from getting the answers you need. You could try keeping a smaller cluster that only gets % percent of the data (eg only 2%) for finding trends, drawing heatmaps etc and that one can run in realtime but your CEO will say that's a stupid idea to your face and it's realtime-all or nothing.
You might say that's a situation you can live with provided the absolutely insane cost savings, but when the company goes at a nicer retreat for only a selective elite few that easily costs 20k, or runs an over the top kitsch open party party/recruiting that costs 80k, and you are being dragged into a meeting on a monday morning and confronted; Why did the "3k per month-3 billion requests per day" system cost 6k this month? (because we had multiple clusters in parallel computing historical data for the past 6 months that you asked for). You just get bitter you didn't give the analytics company the 1 million they asked for and be done with it.