Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reducing BigQuery Costs

blog.peerdb.io

Post reply on HN
Print view View original

11–20 of 40 posts

Re: Reducing BigQuery Costs

#11

Post by ckdarby »

coreyp_1 wrote: post #6

Rule #1. BQ is not a standard database. If you use it like one, it will cost a fortune. Rule #2. BQ is amazing for being able to churn through and analyze massive amounts of data, and can very well be the best option in some use cases. Rule #3. Letting "just anyone" run queries is as dangerous as casually handing a credit card to your drug-addicted cousin. Just wait until you get the bill! Rule #4: Partition and clus…

Use BQ to crunch the larger set into a smaller subset that you need and ram that into PG/MySQL.

Used this to power a +$30M revenue affiliate platform tracking.

Re: Reducing BigQuery Costs

#12

Post by winrid »

boredemployee wrote: post #10
winrid wrote: post #8

Earlier quoted context omitted.

That's nothing. Wait until you incur $150k of Lambda costs in a day!

Did it happen to you? sorry to hear!

Project ended up saving 10x that per year, so wasn't a huge deal. Quickly rewrote it to run as a traditional server, though.

Re: Reducing BigQuery Costs

#13

Post by jojobas »

So far the best solution I've seen to save on BigQuery costs is to deploy your own ClickHouse.

It has its own gotchas and quirks, but cost-wise it's not even comparable.

Re: Reducing BigQuery Costs

#15

Post by ethbr1 »

saisrirampur wrote: post #7
coreyp_1 wrote: post #6

Rule #1. BQ is not a standard database. If you use it like one, it will cost a fortune. Rule #2. BQ is amazing for being able to churn through and analyze massive amounts of data, and can very well be the best option in some use cases. Rule #3. Letting "just anyone" run queries is as dangerous as casually handing a credit card to your drug-addicted cousin. Just wait until you get the bill! Rule #4: Partition and clus…

OP here. 100% agreed on your analysis. Thanks for chiming in. Coming from the Postgres world, this was very counter intuitive for me. I am still not convinced if a database should charge 1000s of $s due to lack of an index (cluster). It could either create the index automatically or explicitly (on the face) warn the user that this can be expensive or else slow.

BigQuery seems to suffer from being overly internal Googly.

A bizarre conversation I witnessed between the BigQuery team and my company at the time (a major customer):

Company: "We need to be able to see our consumption and a breakdown of our bill"

BQ team: "Oh, yeah. We can see how that would be useful. We should probably build that..."

Like, this was a GA product without any thought given to self-serve billing visibility.

I realize billing is usually the last thing bolted on, but I'd expect some basics to be in place before the product ships.

Re: Reducing BigQuery Costs

#16

Post by nine_k »

garciasn wrote: post #14

BQ works better and costs less with clustered and partitioned tables. Who would have thought?!

"Any recent graduate can come up with an architecture better than the current one. What takes a senior engineer is to transition to a better architecture while the business is continuing to operate."

Re: Reducing BigQuery Costs

#17

Post by nikau »

stairlane wrote: post #2

We've recently been struggling with BigQuery and various other GCP services (i.e. CloudRun and Pub sub) as it feels like utilizing these services can feel like a minefield of gotcha's. With their documentation, and limits/quotas being spread all over the place. It's given us more problems than solutions thus far; albeit that could very well be our fault. Has anybody else had this experience? Or are we just doing it w…

Yes, try and find the pause button for a push pub sub for example...

Re: Reducing BigQuery Costs

#18

Post by tbragin »

jojobas wrote: post #13

So far the best solution I've seen to save on BigQuery costs is to deploy your own ClickHouse. It has its own gotchas and quirks, but cost-wise it's not even comparable.

Disclaimer: I work at ClickHouse

Thank you! Glad it helped you.

ClickHouse is in particular well-optimized to take on real-time analytical workloads from BigQuery - both functionally and from a cost perspective.

For anyone wanting to try it yourself, there guide to follow when syncing or migrating workloads from BigQuery to ClickHouse, with some benchmarks and back-of-the-envelope cost comparisons: https://clickhouse.com/blog/clickhouse-bigquery-migrating-da...

Re: Reducing BigQuery Costs

#19

Post by [deleted] »

coreyp_1 wrote: post #6

Rule #1. BQ is not a standard database. If you use it like one, it will cost a fortune. Rule #2. BQ is amazing for being able to churn through and analyze massive amounts of data, and can very well be the best option in some use cases. Rule #3. Letting "just anyone" run queries is as dangerous as casually handing a credit card to your drug-addicted cousin. Just wait until you get the bill! Rule #4: Partition and clus…

[deleted]

Re: Reducing BigQuery Costs

#20

Post by klodolph »

nine_k wrote: post #16
garciasn wrote: post #14

BQ works better and costs less with clustered and partitioned tables. Who would have thought?!

"Any recent graduate can come up with an architecture better than the current one. What takes a senior engineer is to transition to a better architecture while the business is continuing to operate."

And it takes a staff engineer to know which system to refactor.
Post reply on HN