Snowflake’s response to Databricks’ TPC-DS post • Hacker News Board

Re: Snowflake’s response to Databricks’ TPC-DS post

#51

Post by AtlasLion » Sat, Nov 13, 2021, 9:08 AM UTC

throwaway984393 wrote: post #15

"Posting benchmark results is bad because it quickly becomes a race to the wrong solution. But somebody showed us sucking on a benchmark, so here's our benchmark results showing we're better."

Their cofounder was behind vectorwise, which kicked ass in benchmarks, but died as no one even heard of it. You can run the benchmark queries fast, that's great, but can you handle code migrated from vertica? Will you optimiser come up with a good plan for queries built on 15 layers of views? That's what companies in the real world have, not some synthetic benchmark that you can make sure you can run for marketing purposes.

Re: Snowflake’s response to Databricks’ TPC-DS post

#52

Post by cloudbonsai » Sat, Nov 13, 2021, 9:10 AM UTC

michaelhartm wrote: post #29

* Databricks is unethical * Nobody should benchmark anymore, just focus on customers instead * But hey, we just did some benchmarks and we look better than what Databricks claims * Btw, please sign up and do some benchmarks on Snowflake, we actually ship TPC-DS dataset with Snowflake * Btw, we agree with Databricks, let's remove the DeWitt clause, vendors should be able to benchmark each other! * Consistency is more…

The interesting part is that Snowflake omits Databricks' performance scores in their graphs. Here is how they compare on TPC-DS benchmark, based on two companies' self-reports:

* Elapsed time: 3108s (Databricks) vs 3760s (Snowflake)

* Price/Peformance: $242 (Databricks) vs $267 (Snowflake)

Needless to say, these numbers seriously need a verification by independent 3rd parties, but it seems that Databricks is still 18% faster and 10% cheaper than Snowflake?

Re: Snowflake’s response to Databricks’ TPC-DS post

#53

Post by AtlasLion » Sat, Nov 13, 2021, 9:16 AM UTC

The main question I have for DB is, how good is their query optimiser/compiler? It's fun that you can run some predefined set of queries fast. More important is, how good you can run queries in the real world, with suboptimal data models, layers upon layers of badly written views, CTEs, UDFs... That is what matters in the end. Not some synthetic benchmark based on known queries you can optimise specifically for.

Re: Snowflake’s response to Databricks’ TPC-DS post

#54

Post by glogla » Sat, Nov 13, 2021, 9:38 AM UTC

hiyer wrote: post #30

EdwardDiego wrote: post #20

Earlier quoted context omitted.

Very much true. I saw a joke tweet recently something along the lines of - It's amazing how many data engineering scaling issues these days are being solved by just paying Snowflake more money. Spark does take a lot of tuning, but then I'm guessing Databricks offer that service as part of your licensing fee? (I'd hope so if they're selling a product based on FOSS code, there has to be a value add to justify it)

> I'd hope so if they're selling a product based on FOSS code, there has to be a value add to justify it They have some proprietary features like DBIO [1]. They also have some cloud-specific features like storage autoscaling [2] that would not be available in OSS Spark. Even Delta Lake [3] used to be proprietary, but I suspect the rise of open-source frameworks like Iceberg led them to open-source it. Shameless plug…

The open source Delta is not a replacement for the real thing - they did not include features like optimizing small files (small file problem is well known in big data, and much more of a problem once streaming gets involved) and others. It is more of a demo of the real thing. Which does not stop them from repeating everywhere how open they are, of course.

EDIT: the delta also still keeps partitioning information in the hive metastore, while iceberg keeps it in storage, making it a far superior design. Adopting iceberg is harder due to third party tools like AWS Redshift not supporting it - you have to go 100 % of the way.

Re: Snowflake’s response to Databricks’ TPC-DS post

#55

Post by maslam » Sat, Nov 13, 2021, 9:59 AM UTC

AtlasLion wrote: post #53

The main question I have for DB is, how good is their query optimiser/compiler? It's fun that you can run some predefined set of queries fast. More important is, how good you can run queries in the real world, with suboptimal data models, layers upon layers of badly written views, CTEs, UDFs... That is what matters in the end. Not some synthetic benchmark based on known queries you can optimise specifically for.

@AtlasLion you are right real world performance matters. We test extensively with actual workloads, and the speed up holds there too. For example: lots of real world BI queries are repeated over smallish data sets of 10 to 50 GB. We test that size factor and pattern all the time.

Re: Snowflake’s response to Databricks’ TPC-DS post

#56

Post by bjornsing » Sat, Nov 13, 2021, 10:00 AM UTC

rxin wrote: post #26

bjornsing wrote: post #24

> At the end of the script, the overall elapsed time and the geometric mean for all the queries is computed directly by querying the history view of all TPC-DS statements that have executed on the warehouse. The geometric mean? Really? Feels a lot easier to think in terms of arithmetic mean, and perhaps percentiles.

Geometric mean is commonly used in benchmarks when the workloads consists of queries that have large (often orders of magnitude) differences in runtime. Consider 4 queries. Two run for 1sec, and the other two 1000sec. If we look at arithmetic mean, then we are really only taking into account the large queries. But improving geometric mean would require improving all queries. Note that I'm on the opposite side (Databr…

> But improving geometric mean would require improving all queries.

No. Improving the geometric mean only requires reducing the product of their execution times. So if you can make the two 1 ms queries execute in 0.5 ms at the expense of the two 1000 ms queries taking 1800 ms each then that’s an improvement in terms of geometric mean.

So… kind of QED. The geometric mean is not easy to reason about.

Re: Snowflake’s response to Databricks’ TPC-DS post

#57

Post by tyingq » Sat, Nov 13, 2021, 1:44 PM UTC

aptxkid wrote: post #2

Personally I think it’s a great response and very well written. I didn’t jump on the congrats-Databricks wagon when the result first came out because of the weird front page comparison against snowflake. Both companies are doing great work. Focusing on building a better product for your customer is much more meaningful than making your competitor look bad.

It is well written, but there's some sleight of hand here and there too. Like using your lowest tier product to demonstrate price/performance against a competitor's highest tier. The Snowflake lowest tier doesn't have failover, for example...or compliance features.

Re: Snowflake’s response to Databricks’ TPC-DS post

#58

Post by geoduck14 » Sat, Nov 13, 2021, 2:10 PM UTC

kevindeasis wrote: post #35

pxc wrote: post #16

Can someone ELI5 what Snowflake and Databricks are? I spent a few minutes on the Databricks website once and couldn't really penetrate the marketing jargon. There are also some technical terms I don't know at all, and when I've searched for them, the top results are all more Azure stuff. Like wtf is a datalake?

They are a data warehouse with analytics? So data warehouse as a service in the cloud? So they can collect data from different places like sql, images, etc. I think a better question would be what type of data can't they ingest? Once you have your data i guess you can run some analytics to find out what your data tells you

I'd like to add some points: Ive used Snowflake for several years. Snowflake works with structured and semi-structured data (think spreadsheets and JSON). I've never tried working with pics or videos - and I'm not sure it would make sense to do that.

I've evaluated Databricks. It works with the above mentioned structured and semi-structured data. I also suspect it could process unstructured data. My understanding is that it runs Python (and some others), so you can do any "Python stuff, but in the cloud, and on 1000s of computers"

Re: Snowflake’s response to Databricks’ TPC-DS post

#59

Post by geoduck14 » Sat, Nov 13, 2021, 2:12 PM UTC

cloudbonsai wrote: post #52

michaelhartm wrote: post #29

* Databricks is unethical * Nobody should benchmark anymore, just focus on customers instead * But hey, we just did some benchmarks and we look better than what Databricks claims * Btw, please sign up and do some benchmarks on Snowflake, we actually ship TPC-DS dataset with Snowflake * Btw, we agree with Databricks, let's remove the DeWitt clause, vendors should be able to benchmark each other! * Consistency is more…

The interesting part is that Snowflake omits Databricks' performance scores in their graphs. Here is how they compare on TPC-DS benchmark, based on two companies' self-reports: * Elapsed time: 3108s (Databricks) vs 3760s (Snowflake) * Price/Peformance: $242 (Databricks) vs $267 (Snowflake) Needless to say, these numbers seriously need a verification by independent 3rd parties, but it seems that Databricks is still 18…

The way I read this is: DataBricks benchmarked against us, and they messed it up. Here is hou YOU should evaluate Snowflake performance. And, by the way, it is pretty easy to do it.

Re: Snowflake’s response to Databricks’ TPC-DS post

#60

Post by geoduck14 » Sat, Nov 13, 2021, 2:14 PM UTC

kingkongv2 wrote: post #34

michaelhartm wrote: post #29

* Databricks is unethical * Nobody should benchmark anymore, just focus on customers instead * But hey, we just did some benchmarks and we look better than what Databricks claims * Btw, please sign up and do some benchmarks on Snowflake, we actually ship TPC-DS dataset with Snowflake * Btw, we agree with Databricks, let's remove the DeWitt clause, vendors should be able to benchmark each other! * Consistency is more…

If people have never heard of Databricks, now is the time because a 100 billion company just started a war against them. Great marketing win Databricks.

To be fair, I've been equating Databricks for a month or so. Databricks is coming after Snowflake. Snowflake doesn't care. Snowflake has a pretty solid moat with:

EASY SQL, data sharing (they have a marketplace), simple scaling