Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most companies do not need Snowflake or Databricks

kjhealey.medium.com

Post reply on HN
Print view View original

131–140 of 140 posts

Re: Most companies do not need Snowflake or Databricks

#131

Post by ironchef »

ramesh31 wrote: post #112
ironchef wrote: post #69

Earlier quoted context omitted.

Here was my situation. Occasional queries. Over a couple petabyte of data. Customer facing so response in seconds per SLA but > 95 percent of the time the warehouse isn’t running. Cached queries from within 24 hours which don’t require the warehouse to even spin up. Our snowflake costs were significantly less than an FTE. Would that potentially be a situation which “running your own” doesn’t make sense?

>Would that potentially be a situation which “running your own” doesn’t make sense? Look into datalake architectures. RDBMS based data warehousing is obviously not economical at the petabyte scale. But storing all that data in S3 with Delta Lake/Iceberg format and querying with Spark changes things entirely. You only pay for object storage, and S3 read costs are trivial.

> Look into datalake architectures.

Yup .. comfy with iceberg/delta/hudi

> RDBMS based data warehousing is obviously not economical at the petabyte scale.

I never said it was .. I'm simply responding to "I simply cannot understand how anyone chooses this over running your own Spark clusters with Jupyterlab". I'm trying to help you understand why folks would choose a SaaS over run your own.

> But storing all that data in S3 with Delta Lake/Iceberg format and querying with Spark changes things entirely. You only pay for object storage, and S3 read costs are trivial.

No. You don't just pay for object storage + minor S3 read costs.

You pay for operations You pay for someone setting up conventions You pay to not have to optimize data layouts for streaming writes You pay to not have to discover race conditions in s3 when running multiple spark clusters writing to same delta tables You pay to not have to discover that your partitions/clustering needs have changed based on new data or query patterns

But look .. I get it. You have chosen to optimize for cost structures in one way .. and I've chosen to optimize in a different way. In the past I've done exactly as you've said as well. I think being able to seeking to see _why_ folks may have chosen a different path may help you understand other areas to consider in operations.

Re: Most companies do not need Snowflake or Databricks

#132

Post by chrisjc »

polskibus wrote: post #109
kfk wrote: post #26

Earlier quoted context omitted.

Connectivity was the hardest part, we had to write Python connectors for a variety of ERP systems. We have a 1 server setup where we run duckdb and Python scripts, we monitor and orchestrate with Prefect but you can use one of the many Python orchestration tools. We load the finished data marts to a MS SQL server and users connect to it via PowerBI or Excel.

I don't understand why would you use duckdb as an intermediate step to fill Ms SQL DW. Surely you could just go to Ms SQL directly?

I'm sure DuckDB was used to transform/prepare the data from the ERP extract format to an MSSQL ingestion format.

There are plenty of arguments and reasons why you would use DuckDB to do this esp if you're preparing the data for Analytical/OLAP use-cases.

Perhaps a more relevant question might be why they didn't use DataFactory or some other ETL tool/service. DuckDB is rising the occasion for these kinds of use-cases though.

Re: Most companies do not need Snowflake or Databricks

#133

Post by sanderjd »

hodgesrm wrote: post #126
sanderjd wrote: post #118

Earlier quoted context omitted.

> you pay relatively little if you do relatively little data processing Yep, surprised how few people have been mentioning this here. Compared with running a managed postgres instance 100% of the time, running a snowflake xs warehouse for a few minutes a day can be significantly cheaper.

The reason is that Snowflake charges close to cost for object storage and they compress your data when they put it in. However...that means they need to make it up elsewhere. When you do get around to running queries the markup on compute can be 5-10x (or potentially more) depending on the plan you are using. If you do constant, compute heavy aggregation Snowflake is not the right place to do it.

Yes, but my point is, lots of small businesses come out ahead on this.

Which is totally by design! Their whole model is that once it is no longer cost effective, it isn't worth the switching costs to leave. (Exactly like the AWS model, un-coincidentally.)

But I think lots of commenters here (at least at the time I wrote my original comment) seem to be missing that this isn't nefarious or anything, it's win-win for a lot of businesses.

Re: Most companies do not need Snowflake or Databricks

#134

Post by sanderjd »

agent281 wrote: post #130
sanderjd wrote: post #129

Earlier quoted context omitted.

I guess I'll say what I think. I do think it is targeted at that smallest 80% of companies with some digital footprint, and also at most of the top 20%. Or more specifically, I think maybe it's targeted at like the 5th percentile to the 99th percentile. That bottom 5% probably just needs Excel, and that top 1% is probably writing or heavily modifying all their own tools. But I'm not sure the advice is very good from…

I agree. Really seasoned data people are not common enough. Small companies need to buy services to lighten the load. We both seem have a sense of the size of companies at different percentiles. At what percentile would you put your company with petabytes of data?

Super hard to say, so ... 80th or 90th? With very low confidence.

But I do have very high confidence that the 99th percentile is much larger than petabytes (think: what's next after "exa"), and I believe that many companies these days crack into "peta" territory.

But as I saw another comment mention, I think another, probably more important, consideration besides size in bytes is cardinality and structure. So maybe this whole classification we're doing is kind of beside the point :)

Re: Most companies do not need Snowflake or Databricks

#135

Post by agent281 »

sanderjd wrote: post #134
agent281 wrote: post #130

Earlier quoted context omitted.

I agree. Really seasoned data people are not common enough. Small companies need to buy services to lighten the load. We both seem have a sense of the size of companies at different percentiles. At what percentile would you put your company with petabytes of data?

Super hard to say, so ... 80th or 90th? With very low confidence. But I do have very high confidence that the 99th percentile is much larger than petabytes (think: what's next after "exa"), and I believe that many companies these days crack into "peta" territory. But as I saw another comment mention, I think another, probably more important, consideration besides size in bytes is cardinality and structure. So maybe t…

Yeah, it's hard to say with any certainty. I agree that the far end is the curve probably looks nothing like the "neighborhood" a couple percent away, relatively speaking.

I also agree that the variety of data plays a big part in its complexity. If you have a few petabytes of data, but it's really only a handful of tables you can real hone in on the relationships. If it's a wide array of sources with many tables between them then you have some nasty problems like entity resolution.

All happy data sets are alike; each unhappy data set is unhappy in its own way.

Re: Most companies do not need Snowflake or Databricks

#136

Post by tjhunter »

This article has valid points but does not understand the perspective of companies. Companies do not buy technologies. Companies buy solutions.

- Companies do not buy Spark, they buy the ability to process their data and to have multiple personas collaborate (data scientists, data engineers, ...)

- You can do it yourself. It will be cheaper but it will require time, expertise and money, all things that companies do not give easily

- Snowflake and Databricks are elastic: you can start small and grow as you need. This is much easier than justifying the upfront cost of hiring specialized people or asking for trust that your ad-hoc solution will respect whatever enterprise governance rules

(disclaimer: I worked at Databricks for 6 years and talked to hundreds of prospect and actual Databricks users and customers)

Re: Most companies do not need Snowflake or Databricks

#137

Post by sanderjd »

agent281 wrote: post #135
sanderjd wrote: post #134

Earlier quoted context omitted.

Super hard to say, so ... 80th or 90th? With very low confidence. But I do have very high confidence that the 99th percentile is much larger than petabytes (think: what's next after "exa"), and I believe that many companies these days crack into "peta" territory. But as I saw another comment mention, I think another, probably more important, consideration besides size in bytes is cardinality and structure. So maybe t…

Yeah, it's hard to say with any certainty. I agree that the far end is the curve probably looks nothing like the "neighborhood" a couple percent away, relatively speaking. I also agree that the variety of data plays a big part in its complexity. If you have a few petabytes of data, but it's really only a handful of tables you can real hone in on the relationships. If it's a wide array of sources with many tables betw…

> All happy data sets are alike; each unhappy data set is unhappy in its own way.

Ha, gonna steal that for some doc I write someday :)

Re: Most companies do not need Snowflake or Databricks

#138

Post by agent281 »

sanderjd wrote: post #137
agent281 wrote: post #135

Earlier quoted context omitted.

Yeah, it's hard to say with any certainty. I agree that the far end is the curve probably looks nothing like the "neighborhood" a couple percent away, relatively speaking. I also agree that the variety of data plays a big part in its complexity. If you have a few petabytes of data, but it's really only a handful of tables you can real hone in on the relationships. If it's a wide array of sources with many tables betw…

> All happy data sets are alike; each unhappy data set is unhappy in its own way. Ha, gonna steal that for some doc I write someday :)

That's only fair: I stole it from Anna Karenina. :]

https://en.m.wikipedia.org/wiki/Anna_Karenina_principle#:~:t....

Re: Most companies do not need Snowflake or Databricks

#139

Post by sanderjd »

agent281 wrote: post #138
sanderjd wrote: post #137

Earlier quoted context omitted.

> All happy data sets are alike; each unhappy data set is unhappy in its own way. Ha, gonna steal that for some doc I write someday :)

That's only fair: I stole it from Anna Karenina. :] https://en.m.wikipedia.org/wiki/Anna_Karenina_principle#:~:t... .

Ha I know, I love that opener, despite it being super cliche to love it. Things are usually cliches for a good reason :)

Re: Most companies do not need Snowflake or Databricks

#140

Post by kfk »

chrisjc wrote: post #132
polskibus wrote: post #109

Earlier quoted context omitted.

I don't understand why would you use duckdb as an intermediate step to fill Ms SQL DW. Surely you could just go to Ms SQL directly?

I'm sure DuckDB was used to transform/prepare the data from the ERP extract format to an MSSQL ingestion format. There are plenty of arguments and reasons why you would use DuckDB to do this esp if you're preparing the data for Analytical/OLAP use-cases. Perhaps a more relevant question might be why they didn't use DataFactory or some other ETL tool/service. DuckDB is rising the occasion for these kinds of use-cases…

Let me give you some bullets on DataFactory (DF) because it is a question I get a lot.

- DF is quite hard to operationalize, the logging is not so good, Python stack traces are easier to debug and logging can get as detailed as needed

- DF lacks connectors for data ingestion, this is easy in Python as on average a custom connector takes a week or two to develop

- DF is not data pipelines as code and it is becoming really hard to manage governance and change management on UI based ETL tools

- It is hard to enforce best practices on DF. We are finding it is easy to enforce standard ways of writing and managing SQL models and metadata with a combo of dbt and a dbt-ready data catalog

Post reply on HN