Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

medium.com

Post reply on HN
Print view View original

41–50 of 70 posts

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#41

Post by manigandham »

hinkley wrote: post #38
manigandham wrote: post #27

Earlier quoted context omitted.

Is there some magic scale threshold?

Are you asking permission for misbehaving by trying to determine exactly how much is too much? Is there some magic scale threshold for excessive force? No. There is no fine line because you shouldn't be anywhere near the line . 100 billion. It's not 'lots' or millions, or a billion we're talking about here. It's one hundred billion. That's 80 events per day on average for every human they've ever seen (1.2 billion, i…

Nice spin, but you made the statement as if there's a difference so yes, I'm asking you to define the limit. You can't say "anywhere near the line" and not be able to tell me what the line is. Unless you have some understanding for what is reasonable for the business metrics, it's rather useless for any further discussion.

Those ads do have owners, paying a lot of money to both distribute and see how people are interacting with their content so it's the same thing as a website. And there are lots of tracking events to capture so it's easy to add up to billions, but as the article states it's not all user events. Only 10B come from users with everything else being backend server logs.

> I wouldn't be enthusiastic. I might make excuses that I was too busy doing other things to help

Ok... so being dishonest to avoid doing your job is fine? If you are against then don't do it, but what is the point you're trying to make here?

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#42

Post by vgt »

buremba wrote: post #33
vgt wrote: post #16

Earlier quoted context omitted.

BigQuery PM here. I'd love to genuinely understand why you have that impression. BigQuery's on-demand model charges you EXACTLY for what you consume. Meaning, your resource efficiency is 100% [0]. By contrast, typical "cluster pricing" technologies require you to pay for 100% of your cluster uptime. In private data centers, it's difficult to get above 30% average efficiency. BigQuery also takes care of all software,…

BigQuery charges you for the storage so it's not just the queries. It also charges for the streaming inserts which is also not that cheap for large volumes of data. I might be biased since I'm running an analytics company and this is my core business but here's our alternative: The query and storage layer should be separated because often the expensive one the is the query layer. We have API servers which ingest the…

Thanks, can we continue this discussion? Super interesting to me.

One clarification. BigQuery has two methods of ingest. The Streaming API you mention does carry additional cost, but for the added benefit of having your data appear in BigQuery in real-time.

Batch load, however, is entirely free, as in - it doesn't use your "query capacity", and we don't charge for it. I am rather certain this is a very compelling offering. Batch loads encode your data, replicate it, secure it, convert it into our format, and fix any issues your files may have, like optimal file sizes, optimal number of files, optimal file groupings for your queries - lots of subtle things that either slow you down or cause you headaches otherwise. We also manage all the metadata for you (your MySQL instance). We also burn a good chunk of resources post-load re-materializing and optimizing your dataset. We also maintain upgrades, downtime, and so on.

It sounds like your use case is micro-batch, so perhaps you could benefit from our free batch loads?

It also sounds like you don't mind the extra operational overhead, and would rather operate your own stack, run your own upgrades, fix your own file issues, and so on. This is a personal preference, I agree. Customers who prefer our model love the ease of use and would rather focus their energy elsewhere. It is a stated goal of BigQuery to abstract away complexity and make BigQuery as easy to use as possible for everyone, and if we're failing somewhere, we certainly want to know :)

Finally, if I had to equate BigQuery's on-demand pricing, it gives you the ability to go from 0 cores to thousands and back to 0 in sub-second intervals, scoped down to individual query size. This is exactly what's happening under the hood, but all that is abstracted away behind a "run query" button.

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#43

Post by manigandham »

throwaway66666 wrote: post #20

You 'd be surprised how good modern hardware is. We are being hit with between 1.5 and 2 million requests per minute (steady traffic no spikes except increased usage in the weekends), and our analytics solution runs on just 2 main servers and costs $3k per month (total associated costs except human labour which is 1 engineer working on it part time now). We talked with a famous analytics company and they gave us a qu…

How much data size are you dealing with? For that price range, you can probably just get MemSQL or Clickhouse and handle real-time queries across all of the data.

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#44

Post by buremba »

vgt wrote: post #42
buremba wrote: post #33

Earlier quoted context omitted.

BigQuery charges you for the storage so it's not just the queries. It also charges for the streaming inserts which is also not that cheap for large volumes of data. I might be biased since I'm running an analytics company and this is my core business but here's our alternative: The query and storage layer should be separated because often the expensive one the is the query layer. We have API servers which ingest the…

Thanks, can we continue this discussion? Super interesting to me. One clarification. BigQuery has two methods of ingest. The Streaming API you mention does carry additional cost, but for the added benefit of having your data appear in BigQuery in real-time. Batch load, however, is entirely free, as in - it doesn't use your "query capacity", and we don't charge for it. I am rather certain this is a very compelling off…

Our system is also near-realtime. The default batch duration is 3 minutes, it's also configurable but it will also affect the background processes (shard compaction, data organization jobs etc.) so our customers usually prefer 5 minutes which is also near real-time for 10B events per month.

Please note that this is our core business. Of course, we maintain these services but we also try hard to push more work to cloud providers. For example, both AWS and GC offers managed Mysql servers, auto scale groups for nodes, object stores such as S3 and Cloud Storage so we actually maintain the software and help the customers to upgrade the software when they need.

I agree that BigQuery will save time if the users are not familiar with distributed systems and big-data but again, "ability to go from 0 cores to thousands" doesn't make sense to me because in practice I have never experienced such case. Since we need to start the instances it may take up to 2 - 3 minutes but this is often acceptable for data analysts.

Could you please elaborate the part "re-materializing and optimizing your dataset."? We do a number of optimizations for compacting ORC files, bucketing etc. but I would love to hear how BigQuery does post-processing.

As a note, I usually tend to simplify things but most of the BigQuery customers that I see usually do overengineering because of the cost optimization. For example in the article the author uses Redshift for the dashboard data the solution they use for moving data from BigQuery to Redshift also needs to be maintained and it's not that easy. If I'm going to adopt my whole system to the way how BigQuery works and push hard to save costs, then I expect it to be pretty cheap but 40K for reserved slots doesn't sound like cheap to me. We maintain similar size clusters for 20% of this price for the same data volume and it's much more flexible.

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#45

Post by hinkley »

manigandham wrote: post #41
hinkley wrote: post #38

Earlier quoted context omitted.

Are you asking permission for misbehaving by trying to determine exactly how much is too much? Is there some magic scale threshold for excessive force? No. There is no fine line because you shouldn't be anywhere near the line . 100 billion. It's not 'lots' or millions, or a billion we're talking about here. It's one hundred billion. That's 80 events per day on average for every human they've ever seen (1.2 billion, i…

Nice spin, but you made the statement as if there's a difference so yes, I'm asking you to define the limit. You can't say "anywhere near the line" and not be able to tell me what the line is. Unless you have some understanding for what is reasonable for the business metrics, it's rather useless for any further discussion. Those ads do have owners, paying a lot of money to both distribute and see how people are inter…

Whatever let’s you sleep at night, man.

    Ok... so being dishonest to avoid doing your job is fine?
You’d be a hoot as a guest speaker in an ethics class.

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#46

Post by kasey_junk »

throwaway66666 wrote: post #20

You 'd be surprised how good modern hardware is. We are being hit with between 1.5 and 2 million requests per minute (steady traffic no spikes except increased usage in the weekends), and our analytics solution runs on just 2 main servers and costs $3k per month (total associated costs except human labour which is 1 engineer working on it part time now). We talked with a famous analytics company and they gave us a qu…

For those bad at arithmetic that’s 33k rps

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#47

Post by manigandham »

hinkley wrote: post #45
manigandham wrote: post #41

Earlier quoted context omitted.

Nice spin, but you made the statement as if there's a difference so yes, I'm asking you to define the limit. You can't say "anywhere near the line" and not be able to tell me what the line is. Unless you have some understanding for what is reasonable for the business metrics, it's rather useless for any further discussion. Those ads do have owners, paying a lot of money to both distribute and see how people are inter…

Whatever let’s you sleep at night, man. Ok... so being dishonest to avoid doing your job is fine? You’d be a hoot as a guest speaker in an ethics class.

You haven't answered any of the questions or provided any reasonable details.

What does ethics class have to do with this? I'm sure the question of whether lying in your job to avoid doing a particular project would be far more interesting to study anyway, I'll be sure to bring it up the next time I teach one.

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#48

Post by manigandham »

buremba wrote: post #44
vgt wrote: post #42

Earlier quoted context omitted.

Thanks, can we continue this discussion? Super interesting to me. One clarification. BigQuery has two methods of ingest. The Streaming API you mention does carry additional cost, but for the added benefit of having your data appear in BigQuery in real-time. Batch load, however, is entirely free, as in - it doesn't use your "query capacity", and we don't charge for it. I am rather certain this is a very compelling off…

Our system is also near-realtime. The default batch duration is 3 minutes, it's also configurable but it will also affect the background processes (shard compaction, data organization jobs etc.) so our customers usually prefer 5 minutes which is also near real-time for 10B events per month. Please note that this is our core business. Of course, we maintain these services but we also try hard to push more work to clou…

Have you looked at Snowflake Data? They have a similar pay-per-compute setup as your Presto cluster with automatic pause/resume to save any idle time. Good middleground between traditional Redshift and flexible BigQuery.

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#49

Post by GrandNewbien »

maxnevermind wrote: post #37

"In digital advertising, ..." Stopped reading right there. Maybe a great article though. I started working on that thing called BigData not that long time ago but now realized that 50% of the job is advertisement, not fan of it at all, I want to like the end product or at least be neutral about it.

Could you elaborate? There's plenty of big data tasks and roles that aren't even closely related to marketing.

Re: Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery

#50

Post by cheriot »

tzahola wrote: post #17
Arzh wrote: post #12

Earlier quoted context omitted.

I want to know how people use my website, fuck me right?

As soon as the IP packets leave your server, it’s not “your website” anymore.

Would you say as soon as the IP packets leave my device it's not "my data" anymore? It's simplified too far.
Post reply on HN