Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Snowflake’s response to Databricks’ TPC-DS post

snowflake.com

Post reply on HN
Print view View original

31–40 of 111 posts

Re: Snowflake’s response to Databricks’ TPC-DS post

#31

Post by jeffreygoesto »

pxc wrote: post #16

Can someone ELI5 what Snowflake and Databricks are? I spent a few minutes on the Databricks website once and couldn't really penetrate the marketing jargon. There are also some technical terms I don't know at all, and when I've searched for them, the top results are all more Azure stuff. Like wtf is a datalake?

People who downvoted this, please take a minute and reflect that your world is not the whole world. There is a serious question in this comment and there are myriads of topics _you_ have no clue about.

Re: Snowflake’s response to Databricks’ TPC-DS post

#32

Post by hiyer »

rdeboo wrote: post #28
hiyer wrote: post #13

Performance is only one part of the story. The major advantage Snowflake (and to some extent Presto/Trino) brings to the table is it's pretty much plug and play. Spark OTOH usually requires a lot of tweaking to work reliably for your workloads.

I think the comparison was Snowflake vs Databricks SQL. Databricks SQL is a PaaS service just like Snowflake. Also, it uses their Photon engine, which is a proprietary engine written in C++. It is not Spark.

I'm aware that Databricks is a PaaS service, but what Databricks runs under the hood is Spark (with a few proprietary extensions). So your jobs/queries do require some tuning just like with OS Spark.

Spark has had SQL engines (SparkSQL/Hive on Spark) for a long time. Photon is just a new, faster one. Photon tasks also run on Spark executors only, so it's not independent of Spark[1]. Also, while it's proprietary now, I wouldn't be surprised if Databricks open-sources it in the future, like they did with Delta Lake.

1. https://databricks.com/blog/2021/06/17/announcing-photon-pub...

Re: Snowflake’s response to Databricks’ TPC-DS post

#33

Post by jiggawatts »

Spivak wrote: post #9
jiggawatts wrote: post #7

Earlier quoted context omitted.

Audited how? If you look at the Snowflake response the numbers being posted by Databricks look outright faked or otherwise false.

The results are so crazy different that either Snowflake or Databricks are wrong or outright lying.

This is my point also, and I'm being downvoted for it.

If two people are in disagreement about the same facts, then one of them is either misinformed or lying. It's that simple.

If the only recourse seems to be to sink to the level of mud-slinging, with no clear ability to point to the audit trail and say "this is where it all went wrong", then it calls into question the value of that auditing process.

I'm personally unimpressed with the TPC process in general. I remember one "benchmark" that showed the performance of a 2RU server breaking some record, and it was a minor footnote that it was using a disk array with 7,500 drives in it -- dedicated to that one server for the duration of the test. That's an absurd setup that will never exist at any customer, ever.

I ran that same software myself on literally the exact same server, and it couldn't even begin to approach the posted TPC numbers on typical storage. It was at least two orders of magnitude slower.

The rub was that its inefficient usage of storage was the main problem, and the vendor was pulling a smoke & mirrors trick to hide this deficiency of their product. The TPC numbers were an outright fraud in this case, at least in my mind.

So to me, TPC looks like a staged show where the auditors are more like the referees in a WWE wrestling competition.

Re: Snowflake’s response to Databricks’ TPC-DS post

#34

Post by kingkongv2 »

michaelhartm wrote: post #29

* Databricks is unethical * Nobody should benchmark anymore, just focus on customers instead * But hey, we just did some benchmarks and we look better than what Databricks claims * Btw, please sign up and do some benchmarks on Snowflake, we actually ship TPC-DS dataset with Snowflake * Btw, we agree with Databricks, let's remove the DeWitt clause, vendors should be able to benchmark each other! * Consistency is more…

If people have never heard of Databricks, now is the time because a 100 billion company just started a war against them. Great marketing win Databricks.

Re: Snowflake’s response to Databricks’ TPC-DS post

#35

Post by kevindeasis »

pxc wrote: post #16

Can someone ELI5 what Snowflake and Databricks are? I spent a few minutes on the Databricks website once and couldn't really penetrate the marketing jargon. There are also some technical terms I don't know at all, and when I've searched for them, the top results are all more Azure stuff. Like wtf is a datalake?

They are a data warehouse with analytics? So data warehouse as a service in the cloud?

So they can collect data from different places like sql, images, etc. I think a better question would be what type of data can't they ingest?

Once you have your data i guess you can run some analytics to find out what your data tells you

Re: Snowflake’s response to Databricks’ TPC-DS post

#36

Post by mping »

pxc wrote: post #16

Can someone ELI5 what Snowflake and Databricks are? I spent a few minutes on the Databricks website once and couldn't really penetrate the marketing jargon. There are also some technical terms I don't know at all, and when I've searched for them, the top results are all more Azure stuff. Like wtf is a datalake?

A data lake is a system designed for ingesting, and possibly transforming lots of data, a "lake" where you dump your data. This is different from an eg postgres db (a single source of truth for a crud app for example), because it captures more data (eg events) and it's normally not consistent with the single source of truth (the data may arrive in batches, imported from other database, etc). Because the volume of data is normally huge, you need a cluster to store it, and some way of querying it.

Snowflake and data bricks are companies that operate in this space, providing ways to ingest, transform and analyze large volumes of data.

Re: Snowflake’s response to Databricks’ TPC-DS post

#37

Post by jiggawatts »

rxin wrote: post #17
jiggawatts wrote: post #7

Earlier quoted context omitted.

Audited how? If you look at the Snowflake response the numbers being posted by Databricks look outright faked or otherwise false.

There's an official TPC process to audit and review the benchmark process. This debate can be easiest settled by everybody participating in the official benchmark, like we (Databricks) did. The official review process is significantly more complicated than just offering a static dataset that's been highly optimized for answering the exact set of queries. It includes data loading, data maintenance (insert and delete d…

Please also refer to my comment below on the value of the TPC audit process: https://news.ycombinator.com/item?id=29208172

Re: Snowflake’s response to Databricks’ TPC-DS post

#38

Post by _dark_matter_ »

rxin wrote: post #17
jiggawatts wrote: post #7

Earlier quoted context omitted.

Audited how? If you look at the Snowflake response the numbers being posted by Databricks look outright faked or otherwise false.

There's an official TPC process to audit and review the benchmark process. This debate can be easiest settled by everybody participating in the official benchmark, like we (Databricks) did. The official review process is significantly more complicated than just offering a static dataset that's been highly optimized for answering the exact set of queries. It includes data loading, data maintenance (insert and delete d…

Thanks for the additional context here. As someone who works for a company that pays for both databricks and snowflake, I will say that these results don't surprise me.

Spark has always been infinitely configurable, in my experience. There are probably tens of thousands of possible configurations; everything from Java heap size to parquet block size.

Snowflake is the opposite: you can't even specify partitions! There is only clustering.

For a business, running snowflake is easy because engineers don't have to babysit it, and we like it because now we're free to work on more interesting problems. Everybody wins.

Unless those problems are DB optimization. Then snowflake can actually get in your way.

Re: Snowflake’s response to Databricks’ TPC-DS post

#39

Post by ngc248 »

pxc wrote: post #16

Can someone ELI5 what Snowflake and Databricks are? I spent a few minutes on the Databricks website once and couldn't really penetrate the marketing jargon. There are also some technical terms I don't know at all, and when I've searched for them, the top results are all more Azure stuff. Like wtf is a datalake?

A data lake is a company wide data repository. All the "data streams" from all of the different departments will flow into the data lake. Aim is to use this data to get both macro and micro insights.

Re: Snowflake’s response to Databricks’ TPC-DS post

#40

Post by rxin »

_dark_matter_ wrote: post #38
rxin wrote: post #17

Earlier quoted context omitted.

There's an official TPC process to audit and review the benchmark process. This debate can be easiest settled by everybody participating in the official benchmark, like we (Databricks) did. The official review process is significantly more complicated than just offering a static dataset that's been highly optimized for answering the exact set of queries. It includes data loading, data maintenance (insert and delete d…

Thanks for the additional context here. As someone who works for a company that pays for both databricks and snowflake, I will say that these results don't surprise me. Spark has always been infinitely configurable, in my experience. There are probably tens of thousands of possible configurations; everything from Java heap size to parquet block size. Snowflake is the opposite: you can't even specify partitions! There…

Totally. Simplicity is critical. That’s why we built Databricks SQL not based on Spark.

As a matter of fact, we took the extreme approach of not allowing customers (or ourselves) to set any of the known knobs. We want to force ourselves to build the best the system to run well out of the box and yet still beats data warehouses in price perf. The official result involved no tuning. It was partitioned by date, loaded data in, provisioned a Databricks SQL endpoint and that’s it. No additional knobs or settings. (As a matter of fact, Snowflakes own sample TPC-DS dataset has more tuning than the ones we did. They clustered by multiple columns specifically to optimize for the exact set of queries.)

Post reply on HN