Snowflake’s response to Databricks’ TPC-DS post • Hacker News Board

Re: Snowflake’s response to Databricks’ TPC-DS post

#61

Post by geoduck14 » Sat, Nov 13, 2021, 2:20 PM UTC

rxin wrote: post #40

_dark_matter_ wrote: post #38

Earlier quoted context omitted.

Thanks for the additional context here. As someone who works for a company that pays for both databricks and snowflake, I will say that these results don't surprise me. Spark has always been infinitely configurable, in my experience. There are probably tens of thousands of possible configurations; everything from Java heap size to parquet block size. Snowflake is the opposite: you can't even specify partitions! There…

Totally. Simplicity is critical. That’s why we built Databricks SQL not based on Spark. As a matter of fact, we took the extreme approach of not allowing customers (or ourselves) to set any of the known knobs. We want to force ourselves to build the best the system to run well out of the box and yet still beats data warehouses in price perf. The official result involved no tuning. It was partitioned by date, loaded d…

>That’s why we built Databricks SQL not based on Spark.

Wait... really? The sales folks I've been talking to didn't mention this. I assumed that when I ran SQL inside my Python, it was decomposed into Spark SQL with weird join problems (and other nuances I'm not fully familiar with).

Not that THAT would have changed my mind. But it would have changed the calculus of "who uses this tool at my company" and "who do I get on board with this thing"

Edit: To add, I've been a customer of Snowflake for years. I've been evaluating Databricks for 2 months, and put the POC on hold.

Re: Snowflake’s response to Databricks’ TPC-DS post

#62

Post by dekhn » Sat, Nov 13, 2021, 2:30 PM UTC

jeffreygoesto wrote: post #31

pxc wrote: post #16

Can someone ELI5 what Snowflake and Databricks are? I spent a few minutes on the Databricks website once and couldn't really penetrate the marketing jargon. There are also some technical terms I don't know at all, and when I've searched for them, the top results are all more Azure stuff. Like wtf is a datalake?

People who downvoted this, please take a minute and reflect that your world is not the whole world. There is a serious question in this comment and there are myriads of topics _you_ have no clue about.

sure, but if I see the term 'data lake' I'm gonna Bing it, with the first result being https://aws.amazon.com/big-data/datalakes-and-analytics/what... which explains it nicely.

ELI5 is for reddit, generally here we expect you can google it to get the ELI5 explanation before giving us your hot take in a comment

Re: Snowflake’s response to Databricks’ TPC-DS post

#63

Post by dekhn » Sat, Nov 13, 2021, 2:32 PM UTC

cmhill wrote: post #12

mst wrote: post #11

Earlier quoted context omitted.

Databricks and snowflake should pay an independent third party to re-run these. In-house benchmarks by either company don't count with results this different.

Databricks didn't run the Snowflake comparison in-house. From their article it says: "These results were corroborated by research from Barcelona Supercomputing Center, which frequently runs TPC-DS on popular data warehouses. Their latest research benchmarked Databricks and Snowflake, and found that Databricks was 2.7x faster and 12x better in terms of price performance."

I don't trust a supercomputer center to do a good job running a TPC benchmark (I do trust them to run LINPACK benchmarks).

Re: Snowflake’s response to Databricks’ TPC-DS post

#64

Post by bpaneural » Sat, Nov 13, 2021, 2:35 PM UTC

geoduck14 wrote: post #60

kingkongv2 wrote: post #34

Earlier quoted context omitted.

If people have never heard of Databricks, now is the time because a 100 billion company just started a war against them. Great marketing win Databricks.

To be fair, I've been equating Databricks for a month or so. Databricks is coming after Snowflake. Snowflake doesn't care. Snowflake has a pretty solid moat with: EASY SQL, data sharing (they have a marketplace), simple scaling

You'll need to revisit this again. In the last two years Databricks has built a lead and a bigger moat. They're essentially nice chaps with a huge community backing them. And we all love their open source tools which essentially powers not only their big data platforms, but everyone else's too (AWS, GCP).

Re: Snowflake’s response to Databricks’ TPC-DS post

#65

Post by socaldata » Sat, Nov 13, 2021, 3:08 PM UTC

Take all the problems you have had with data warehousing and throw them in a proprietary cloud. That is Snowflake. They are the best today.

Databricks started with the cloud datalake, sitting natively on parquet and using cloud native tools, fully open. Recently they added SQL to help democratize the data in the data lake versus moving it back and forth into a proprietary data warehouse.

The selling point in Databricks is why move the data around when you can just have it in one place IF performance is the same or better.

This is what led to the latest benchmark which in the writing appears to be unbiased.

In snowflakes response however, they condemn it but then submit their own fundings. Sound a lot lot trump telling everyone he had billions of people attend his inauguration, doesn’t it?

Anyhow, I trust independent studies more than I do coming from vendors. It cannot be argued or debated unless it was unfairly done. I think we are all smart enough to be careful with studies of any kind, but I can see why Databricks was excited about the findings.

Re: Snowflake’s response to Databricks’ TPC-DS post

#66

Post by uvdn7 » Sat, Nov 13, 2021, 3:17 PM UTC

socaldata wrote: post #65

Take all the problems you have had with data warehousing and throw them in a proprietary cloud. That is Snowflake. They are the best today. Databricks started with the cloud datalake, sitting natively on parquet and using cloud native tools, fully open. Recently they added SQL to help democratize the data in the data lake versus moving it back and forth into a proprietary data warehouse. The selling point in Databric…

Whose result can be trusted is beside the point - I actually believe both experiments were likely conducted in good faith but with incomplete context. But that’s beside the point. The point is there’s no good reason to start a benchmark war to begin with.

Re: Snowflake’s response to Databricks’ TPC-DS post

#67

Post by pxc » Sat, Nov 13, 2021, 3:22 PM UTC

dekhn wrote: post #62

jeffreygoesto wrote: post #31

Earlier quoted context omitted.

People who downvoted this, please take a minute and reflect that your world is not the whole world. There is a serious question in this comment and there are myriads of topics _you_ have no clue about.

sure, but if I see the term 'data lake' I'm gonna Bing it, with the first result being https://aws.amazon.com/big-data/datalakes-and-analytics/what... which explains it nicely. ELI5 is for reddit, generally here we expect you can google it to get the ELI5 explanation before giving us your hot take in a comment

Yeah, that's exactly the kind of content I found unsuitable when I did a web search for the term. It spends a whole two sentences giving an explanation that tells me very little about how data lakes are anything more specific than a cloud-hosted database solution, and moves on to

> Organizations that successfully generate business value from their data, will outperform their peers.

at which point I'm like

> ok, I'm reading a covert advertisement about Fancy Cloud Technology aimed at some kind of big-spending manager, which is unlikely to tell me meaningfully what this actually is

and I'm out. I was looking for content that was in a more neutral, purely educational genre, and wondering what collection of non-cloud analogues it replaces/is composed of. Someone writing in the comments

> I used it to transform several terabytes of JSON into nice relational data for analysts without too much effort

is way, way more direct and helpful than mentioning that 'unlike data warehouses, data lakes support non-relational data'. Like great, it's a cloud thing that supports a variety of databases. But what is it?

> before giving us your hot take in a comment

I didn't give any take at all? I just really found all the sources that came up on the first page of search results to be almost in the wrong genre for me, and expected (correctly) that people on this site would be able to produce descriptions in 1-5 sentences that worked way better for me.

Pretty much all of the answers I got here were really good, and I'm glad I asked.

Re: Snowflake’s response to Databricks’ TPC-DS post

#68

Post by uvdn7 » Sat, Nov 13, 2021, 3:27 PM UTC

dekhn wrote: post #62

jeffreygoesto wrote: post #31

Earlier quoted context omitted.

People who downvoted this, please take a minute and reflect that your world is not the whole world. There is a serious question in this comment and there are myriads of topics _you_ have no clue about.

sure, but if I see the term 'data lake' I'm gonna Bing it, with the first result being https://aws.amazon.com/big-data/datalakes-and-analytics/what... which explains it nicely. ELI5 is for reddit, generally here we expect you can google it to get the ELI5 explanation before giving us your hot take in a comment

It’s probably just me but the distinction between datalake and data warehouse seems like splitting hairs. Unstructured data can always be stored on structure databases. What’s the main reason for both to coexist?

Re: Snowflake’s response to Databricks’ TPC-DS post

#69

Post by uvdn7 » Sat, Nov 13, 2021, 3:30 PM UTC

michaelhartm wrote: post #29

* Databricks is unethical * Nobody should benchmark anymore, just focus on customers instead * But hey, we just did some benchmarks and we look better than what Databricks claims * Btw, please sign up and do some benchmarks on Snowflake, we actually ship TPC-DS dataset with Snowflake * Btw, we agree with Databricks, let's remove the DeWitt clause, vendors should be able to benchmark each other! * Consistency is more…

I don’t think they are saying benchmark is not important but rather public benchmark war being a distraction.

Re: Snowflake’s response to Databricks’ TPC-DS post

#70

Post by uvdn7 » Sat, Nov 13, 2021, 3:33 PM UTC

rxin wrote: post #17

jiggawatts wrote: post #7

Earlier quoted context omitted.

Audited how? If you look at the Snowflake response the numbers being posted by Databricks look outright faked or otherwise false.

There's an official TPC process to audit and review the benchmark process. This debate can be easiest settled by everybody participating in the official benchmark, like we (Databricks) did. The official review process is significantly more complicated than just offering a static dataset that's been highly optimized for answering the exact set of queries. It includes data loading, data maintenance (insert and delete d…

Snowflake claims the snowflake result from Databricks was not audited. It’s not that Databricks numbers were artificially good but rather Snowflake’s number was unreasonably bad.