We built a modern data stack from scratch and reduced our bill by 70%
1–10 of 60 posts
Re: We built a modern data stack from scratch and reduced our bill by 70%
#2Good read.
I do have a question on the BigQuery. i f you were experiencing unpredictable query costs or customization issues, that sounds like user error. There are ways to optimize or commit slots for reducing the cost. Did you try that ?
Re: We built a modern data stack from scratch and reduced our bill by 70%
#3Good read. I do have a question on the BigQuery. i f you were experiencing unpredictable query costs or customization issues, that sounds like user error. There are ways to optimize or commit slots for reducing the cost. Did you try that ?
As for BigQuery, while it's a great tool, we faced challenges with high-volume, small queries where costs became unpredictable as it is priced per data volume scanned. Clustered tables, Materialised views helped to some extent, but they didn’t fully mitigate the overhead for our specific workloads. There are ways to overcome and optimize it for sure so i wouldn't exactly put it on GBQ or any limitations.
It’s always a trade-off, and we made the call that best fit our scale, workloads, and long-term plans
Re: We built a modern data stack from scratch and reduced our bill by 70%
#4AKA The Monty Hall Rewrite https://alexsexton.com/blog/2014/11/the-monty-hall-rewrite
Re: We built a modern data stack from scratch and reduced our bill by 70%
#5Good read. I do have a question on the BigQuery. i f you were experiencing unpredictable query costs or customization issues, that sounds like user error. There are ways to optimize or commit slots for reducing the cost. Did you try that ?
As for BigQuery, while it's a great tool, we faced challenges with high-volume, small queries where costs became unpredictable as it is priced per data volume scanned. Clustered tables, Materialised views helped to some extent, but they didn’t fully mitigate the overhead for our specific workloads. There are ways to overcome and optimize it for sure so i wouldn't exactly put it on GBQ or any limitations. It’s always…
Hmm, Okay.
I am not sure if managing kafka connect cluster in too expensive in long term. This solution might work for you based on your needs. I would suggest to look for alternatives.
Re: We built a modern data stack from scratch and reduced our bill by 70%
#6We did have a discussion on Self vs Managed and TCOs associated with it.
1> We have multi regional setup so it came up with Data Sovereignty requirements.
2> Vendor Lock ins - Few of the services were not available in that geographic region
3> With managed services, you often pay for capacity you might not always use. our workloads were often consistent and predictable, so self managed solutions helped in fine tuning our resources.
4> One og the goal was to keep our storage and compute loosely coupled while staying Iceberg-compatible for flexibility. Whether it’s Trino today or Snowflake/Databricks tomorrow, we aren’t locked in.
Re: We built a modern data stack from scratch and reduced our bill by 70%
#7These just seems like over engineered solutions trying to guarantee their job security. When the dataflows are so straight forward, just replicate into pick your OLAP, and transform there.
Re: We built a modern data stack from scratch and reduced our bill by 70%
#8.. how many engineers?
Re: We built a modern data stack from scratch and reduced our bill by 70%
#9Good read. I do have a question on the BigQuery. i f you were experiencing unpredictable query costs or customization issues, that sounds like user error. There are ways to optimize or commit slots for reducing the cost. Did you try that ?
As for BigQuery, while it's a great tool, we faced challenges with high-volume, small queries where costs became unpredictable as it is priced per data volume scanned. Clustered tables, Materialised views helped to some extent, but they didn’t fully mitigate the overhead for our specific workloads. There are ways to overcome and optimize it for sure so i wouldn't exactly put it on GBQ or any limitations. It’s always…
Did you consider slots based pricing model for BQ?
Re: We built a modern data stack from scratch and reduced our bill by 70%
#10So you saved just $20k per year? Not sure the context of your company but I’m not sure if this turns out to be a net win given the cost of engineering resources to produce this infra gain