Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

medium.com

Post reply on HN
Print view View original

61–70 of 70 posts

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#61

Post by manigandham »

buremba wrote: post #57
manigandham wrote: post #48

Earlier quoted context omitted.

Have you looked at Snowflake Data? They have a similar pay-per-compute setup as your Presto cluster with automatic pause/resume to save any idle time. Good middleground between traditional Redshift and flexible BigQuery.

Yes, they have their own query engine but the idea is same; separate storage and query layer, scale the query layer on demand and charge for it. I like their approach but not everyone is willing to depend on a third party company for their company data.

How is that different than BigQuery then? GCP and AWS are also third party companies.

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#62

Post by vgt »

slap_shot wrote: post #59
vgt wrote: post #23

Earlier quoted context omitted.

Genuinely thanks for the info, this is super insightful. I'll argue that BigQuery's per-query pricing charges you JUST for the resources you use (well, data scaned), so it SHOULD be far less expensive than a model that charges you for the luxury of having a cluster sit idle (and often at only 30% utilization), correct? Can you help me unpack this further? I think pay-per-query is ultra-efficient, but difficult to pre…

> ...BigQuery's per-query pricing charges you JUST for the resources you use (well, data scaned), so it SHOULD be far less expensive than a model that charges you for the luxury of having a cluster sit idle (and often at only 30% utilization), correct? > For example, if Netflix charged you a penny per minute of watchtime, you'd have no idea if it's more expensive or less expensive, but you'd be assured it's more effi…

Thanks, this is very valuable feedback, worth in gold. Please feel free to email me directly to unpack our future plans here.

The Netflix analogy also applies if they bill $0.001 per minute - the number I chose is not indicative of reality, I was just making a point. My goal was to demonstrate the fact that it's more efficient per-resource.

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#63

Post by manigandham »

beepbeepbeep1 wrote: post #51
GrandNewbien wrote: post #49

Earlier quoted context omitted.

Could you elaborate? There's plenty of big data tasks and roles that aren't even closely related to marketing.

Not my experince. I'd say the majority of big data is unethical both in capture and use trying to sell you something or sell someone else something on you, a small amount is for general good. I wasn't involved on the project but a major Telecoms company i was at was prototyping scanning for mobile devices on their provided home routers. If they didn't recognise a mibile as being on their network they would then targe…

The vast majority of big data is used today to provide everything you depend on, like email, healthcare, banking, transportation, government services, etc.

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#64

Post by manigandham »

iwintermute wrote: post #53
wetha wrote: post #29

Earlier quoted context omitted.

Around query pricing, my understanding is BigQuery charges by bytes scanned uncompressed. Redshift Spectrum/Athena charges by bytes scanned compressed. That makes Athena/Spectrum cheaper as well.

It's not strictly true - depends on the rate. The only thing you can say for sure - uncompressed analysis on Athena is too expensive

They have the same rate. BigQuery and Athena are $5/TB scanned. Unless you store everything in raw JSON text, Athena will be cheaper even with most basic compression.

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#65

Post by buremba »

manigandham wrote: post #61
buremba wrote: post #57

Earlier quoted context omitted.

Yes, they have their own query engine but the idea is same; separate storage and query layer, scale the query layer on demand and charge for it. I like their approach but not everyone is willing to depend on a third party company for their company data.

How is that different than BigQuery then? GCP and AWS are also third party companies.

The motivation is rather different. If you ask Redshift users, the motivation is mostly that they're already using AWS and depend on their services. That's the case for GCP as well, I would love to hear from the Google side of the story but AWS customers don't usually use BigQuery for their analytics stack because as the author explained it's not that easy. That's why BigQuery is not getting enough attraction from the outside, even though it's a cool technology.

We all agree that Redshift is not scalable at some point and people switch to other technologies when they need to but that's not the point. Creating a Redshift cluster is dead-easy and getting the data in it is also not complex compared to other solutions if you're already an AWS user.

The case for Snowflake is different, they also use AWS but they manage your services. Although they did a smart move and store the data on their customers' S3 buckets in order to make them feel like they "own" the data but it's possible only because AWS lets them to that way.

I believe that AWS doesn't try to make Redshift cost-efficient for large volumes of data because they already have the advantage of vendor-lock and making money from their large enterprise customers processing billions of data points in Redshift. That's why there are many Redshift monitoring startups out there to save the cost for you.

On the other hand, AWS is smart enough to build a Snowflake-like solution when their Redshift users start to switch BigQuery and Snowflake and the companies such as Snowflake needs to be prepared when the day comes. Cloud is killing everyone including us.

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#66

Post by manigandham »

buremba wrote: post #65
manigandham wrote: post #61

Earlier quoted context omitted.

How is that different than BigQuery then? GCP and AWS are also third party companies.

The motivation is rather different. If you ask Redshift users, the motivation is mostly that they're already using AWS and depend on their services. That's the case for GCP as well, I would love to hear from the Google side of the story but AWS customers don't usually use BigQuery for their analytics stack because as the author explained it's not that easy. That's why BigQuery is not getting enough attraction from th…

Snowflake has their own storage unless you opt for the enterprise version to have it stored in your own account. They also use their own table format (like BigQuery) but you can always export it.

It's true that AWS has Redshift Spectrum (and Athena) to help with more scalable querying across S3, however I don't think that makes a big risk for another company that provide a focused offering on top. Snowflake is very well capitalized with close to $500M in investment and plenty of customers so I wouldn't worry about them going out of business.

I especially like them because they have the elastic computing style of BigQuery but charge for computing time rather than data scanned, which is a much more effective billing model than anything else out there.

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#67

Post by manigandham »

slap_shot wrote: post #59
vgt wrote: post #23

Earlier quoted context omitted.

Genuinely thanks for the info, this is super insightful. I'll argue that BigQuery's per-query pricing charges you JUST for the resources you use (well, data scaned), so it SHOULD be far less expensive than a model that charges you for the luxury of having a cluster sit idle (and often at only 30% utilization), correct? Can you help me unpack this further? I think pay-per-query is ultra-efficient, but difficult to pre…

> ...BigQuery's per-query pricing charges you JUST for the resources you use (well, data scaned), so it SHOULD be far less expensive than a model that charges you for the luxury of having a cluster sit idle (and often at only 30% utilization), correct? > For example, if Netflix charged you a penny per minute of watchtime, you'd have no idea if it's more expensive or less expensive, but you'd be assured it's more effi…

To repeat my other comments, the issue is that BQ doesn't actually charge for compute but for data scanned. You already pay for storage (uncompressed but that's a separate issue) so paying again on that metric doesn't make sense for querying since you're using CPU cores, which is a time-based metric. If I use 1000 cores for 10 seconds vs 10 minutes, it makes sense to pay exactly that regardless of how much data was scanned.

We've run into the same issue where being curious with BigQuery actually becomes problematic as users are perfectly fine waiting an extra minute to scan 10TBs, but are afraid of the $50 bill that comes with that, especially for every little query that might be mistyped or out of their hands when using BI tools that run queries of their own.

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#68

Post by buremba »

manigandham wrote: post #66
buremba wrote: post #65

Earlier quoted context omitted.

The motivation is rather different. If you ask Redshift users, the motivation is mostly that they're already using AWS and depend on their services. That's the case for GCP as well, I would love to hear from the Google side of the story but AWS customers don't usually use BigQuery for their analytics stack because as the author explained it's not that easy. That's why BigQuery is not getting enough attraction from th…

Snowflake has their own storage unless you opt for the enterprise version to have it stored in your own account. They also use their own table format (like BigQuery) but you can always export it. It's true that AWS has Redshift Spectrum (and Athena) to help with more scalable querying across S3, however I don't think that makes a big risk for another company that provide a focused offering on top. Snowflake is very w…

What's the focused offering provided by them? Does their system more scalable, efficient or cheap or is there any technical limitations of the AWS products? Personally, I don't think $500M matters if their customers (I'm assuming most of their revenue comes from enterprise customers because that's how it works usually) churn in a few years.

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#69

Post by manigandham »

buremba wrote: post #68
manigandham wrote: post #66

Earlier quoted context omitted.

Snowflake has their own storage unless you opt for the enterprise version to have it stored in your own account. They also use their own table format (like BigQuery) but you can always export it. It's true that AWS has Redshift Spectrum (and Athena) to help with more scalable querying across S3, however I don't think that makes a big risk for another company that provide a focused offering on top. Snowflake is very w…

What's the focused offering provided by them? Does their system more scalable, efficient or cheap or is there any technical limitations of the AWS products? Personally, I don't think $500M matters if their customers (I'm assuming most of their revenue comes from enterprise customers because that's how it works usually) churn in a few years.

A more focused data warehouse product that is priced on a better model and is more scalable and efficient and cheaper than any of the existing options. $500M means they'll be around longer than most of the startup customers that try them.

Anyway, at this point I'm just repeating myself so I suggest you actually try them if you care for a better model. It works for us at 600B rows of data that was too expensive with BigQuery and too slow and complicated with Redshift/Spectrum.

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#70

Post by buremba »

manigandham wrote: post #69
buremba wrote: post #68

Earlier quoted context omitted.

What's the focused offering provided by them? Does their system more scalable, efficient or cheap or is there any technical limitations of the AWS products? Personally, I don't think $500M matters if their customers (I'm assuming most of their revenue comes from enterprise customers because that's how it works usually) churn in a few years.

A more focused data warehouse product that is priced on a better model and is more scalable and efficient and cheaper than any of the existing options. $500M means they'll be around longer than most of the startup customers that try them. Anyway, at this point I'm just repeating myself so I suggest you actually try them if you care for a better model. It works for us at 600B rows of data that was too expensive with B…

Sounds fair. I'm convinced to try out their service, it's good to see that they also added pricing to their website. Let's see how it's better compared to Presto as they have the similar use-case.
Post reply on HN