Live data from Hacker News

Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

medium.com

51–60 of 70 posts

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#51

"In digital advertising, ..." Stopped reading right there. Maybe a great article though. I started working on that thing called BigData not that long time ago but now realized that 50% of the job is advertisement, not fan of it at all, I want to like the end product or at least be neutral about it.

Could you elaborate? There's plenty of big data tasks and roles that aren't even closely related to marketing.

Not my experince. I'd say the majority of big data is unethical both in capture and use trying to sell you something or sell someone else something on you, a small amount is for general good.

I wasn't involved on the project but a major Telecoms company i was at was prototyping scanning for mobile devices on their provided home routers. If they didn't recognise a mibile as being on their network they would then target you with mobile deals. Devs saw the issues with this, essentially scanning your home network from their routers and sending the data back to be processed. Managers found Apple's mac address rotation annoying and lost the point why Apple do it.

Unfortunately besides quitting the job devs on the project don't have a say as they don't have the power or say in the final decisions at these companies. It's easy to say you would quit but it's harder when you have a mortgage, wife and kids for example.

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#52
post #16
post #11

Sounds like they over-engineered the solution. If you have ad-hoc use-case, BigQuery is great but it's quite expensive. If you just need to pre-calculate the metrics using SQL, Athena / Prestodb / Clickhouse / Redshift Spectrum might be much easier and cost-efficient.

BigQuery PM here. I'd love to genuinely understand why you have that impression. BigQuery's on-demand model charges you EXACTLY for what you consume. Meaning, your resource efficiency is 100% [0]. By contrast, typical "cluster pricing" technologies require you to pay for 100% of your cluster uptime. In private data centers, it's difficult to get above 30% average efficiency. BigQuery also takes care of all software,…

> In private data centers, it's difficult to get above 30% average efficiency.

What exactly does that mean?

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#53
post #29
post #16

Earlier quoted context omitted.

BigQuery PM here. I'd love to genuinely understand why you have that impression. BigQuery's on-demand model charges you EXACTLY for what you consume. Meaning, your resource efficiency is 100% [0]. By contrast, typical "cluster pricing" technologies require you to pay for 100% of your cluster uptime. In private data centers, it's difficult to get above 30% average efficiency. BigQuery also takes care of all software,…

Around query pricing, my understanding is BigQuery charges by bytes scanned uncompressed. Redshift Spectrum/Athena charges by bytes scanned compressed. That makes Athena/Spectrum cheaper as well.

It's not strictly true - depends on the rate. The only thing you can say for sure - uncompressed analysis on Athena is too expensive

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#54

"In digital advertising, ..." Stopped reading right there. Maybe a great article though. I started working on that thing called BigData not that long time ago but now realized that 50% of the job is advertisement, not fan of it at all, I want to like the end product or at least be neutral about it.

Could you elaborate? There's plenty of big data tasks and roles that aren't even closely related to marketing.

Maybe it's different in the states, because a lot of tech giants are based there. But in Europe, I feel like, there are banks that inherently have a lot of data and they can use BigData stack for ETL and other stuff, and there is the rest who just want to get some data on users and sell them some products they don't really want, it's not necessary an advisement company by itself, it can be an analytical platform or just a department in some big e-store.

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#55
post #16

Earlier quoted context omitted.

BigQuery PM here. I'd love to genuinely understand why you have that impression. BigQuery's on-demand model charges you EXACTLY for what you consume. Meaning, your resource efficiency is 100% [0]. By contrast, typical "cluster pricing" technologies require you to pay for 100% of your cluster uptime. In private data centers, it's difficult to get above 30% average efficiency. BigQuery also takes care of all software,…

> In private data centers, it's difficult to get above 30% average efficiency. What exactly does that mean?

Typically servers are only under a reasonable workload for load 30% of the day. Resulting in that hardware being unused (wasted) the other 70%.

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#56

Earlier quoted context omitted.

Could you elaborate? There's plenty of big data tasks and roles that aren't even closely related to marketing.

Not my experince. I'd say the majority of big data is unethical both in capture and use trying to sell you something or sell someone else something on you, a small amount is for general good. I wasn't involved on the project but a major Telecoms company i was at was prototyping scanning for mobile devices on their provided home routers. If they didn't recognise a mibile as being on their network they would then targe…

Maybe you'd find Amino interesting. They use big data for healthcare savings.

https://amino.com

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#57
post #44

Earlier quoted context omitted.

Our system is also near-realtime. The default batch duration is 3 minutes, it's also configurable but it will also affect the background processes (shard compaction, data organization jobs etc.) so our customers usually prefer 5 minutes which is also near real-time for 10B events per month. Please note that this is our core business. Of course, we maintain these services but we also try hard to push more work to clou…

Have you looked at Snowflake Data? They have a similar pay-per-compute setup as your Presto cluster with automatic pause/resume to save any idle time. Good middleground between traditional Redshift and flexible BigQuery.

Yes, they have their own query engine but the idea is same; separate storage and query layer, scale the query layer on demand and charge for it. I like their approach but not everyone is willing to depend on a third party company for their company data.

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#58

"In digital advertising, ..." Stopped reading right there. Maybe a great article though. I started working on that thing called BigData not that long time ago but now realized that 50% of the job is advertisement, not fan of it at all, I want to like the end product or at least be neutral about it.

I work for a digital advertising company, handling the terabytes of data we produce each day - and we're not using it to track people or build profiles, we're using it to monitor our system and record important events. Several thousand publishers monetise their sites through us, so I'm happily neutral about the industry.

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#59
post #23

Earlier quoted context omitted.

The answer comes down to one line in your BigQuery Under the Hood article[0]: "The answer is very simple — BigQuery has this much hardware (and much much more) available to devote to your queries for seconds at a time. BigQuery is powered by multiple data centers, each with hundreds of thousands of cores, dozens of petabytes in storage capacity, and terabytes in networking bandwidth. The numbers above — 300 disks, 30…

Genuinely thanks for the info, this is super insightful. I'll argue that BigQuery's per-query pricing charges you JUST for the resources you use (well, data scaned), so it SHOULD be far less expensive than a model that charges you for the luxury of having a cluster sit idle (and often at only 30% utilization), correct? Can you help me unpack this further? I think pay-per-query is ultra-efficient, but difficult to pre…

> ...BigQuery's per-query pricing charges you JUST for the resources you use (well, data scaned), so it SHOULD be far less expensive than a model that charges you for the luxury of having a cluster sit idle (and often at only 30% utilization), correct?

> For example, if Netflix charged you a penny per minute of watchtime, you'd have no idea if it's more expensive or less expensive, but you'd be assured it's more efficient.

The Netflix analogy is actually quite good. Using that, let's say that I have a family of six, I'm billed $0.01 per minute of viewership, but I'm getting the best/quality speed. Each person watches 40 minutes a day, for all 30 days, for a grand total of $72. Far greater than the fixed cost of $10.99 (with, say, significantly less quality, speed, and minutes per month of viewership).

In the real world, the family of six is my team of data analysts, scientists, and BI folks who are querying my database from 9-5 every weekday.

A customer of mine, a large NYC publisher, who you have heard of and probably read, evaluated BigQuery in 2017. They loaded BQ with the exact same data as their Redshift cluster, pointed their Looker instance at it, and in just one day blew through 1/4 their typical Redshift budget. All the queries were faster. Way faster than they needed to be, actually.

Going back to my original post above, the issue here is that just because Google CAN throw these massive amounts of resources at my problem, doesn't mean I can afford to use that level computation for each query, or would even want to. In the Netflix example, I'm happy if Sally and Billy get less quality or limited time watching Netflix, as long as my bill stays at $10.99.

For most companies, their Redshift cluster is optimized to be able to handle their peak workload WELL ENOUGH. That means that queries won't be as fast as BigQuery - and that's totally fine. And it means that the cluster will be underutilized for large portions of the night and weekends - again, totally fine. They just need their usage capped at a predetermined cost, and have their queries finishing in a reasonable amount of time.

I've posed this Google employees before and I'm hit with "well, you can limit how much each person can query a day." Except that isn't a acceptable solution. I can't have analysts sitting around unable to query their database because they've exceeded their daily limit. They'd rather just fire off their Redshift query and if it takes a little bit longer, so be it.

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#60
post #23

Earlier quoted context omitted.

Genuinely thanks for the info, this is super insightful. I'll argue that BigQuery's per-query pricing charges you JUST for the resources you use (well, data scaned), so it SHOULD be far less expensive than a model that charges you for the luxury of having a cluster sit idle (and often at only 30% utilization), correct? Can you help me unpack this further? I think pay-per-query is ultra-efficient, but difficult to pre…

> ...BigQuery's per-query pricing charges you JUST for the resources you use (well, data scaned), so it SHOULD be far less expensive than a model that charges you for the luxury of having a cluster sit idle (and often at only 30% utilization), correct? > For example, if Netflix charged you a penny per minute of watchtime, you'd have no idea if it's more expensive or less expensive, but you'd be assured it's more effi…

Netflix's analogy is great, and I would add: being charged $0.01 per minute of viewership is so stressful you would consume less, or you would be stressed about what your parents would say about the bill.
Post reply on HN