Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Snowflake’s response to Databricks’ TPC-DS post

snowflake.com

Post reply on HN
Print view View original

21–30 of 111 posts

Re: Snowflake’s response to Databricks’ TPC-DS post

#21

Post by rxin »

lmeyerov wrote: post #10
maslam wrote: post #8

Earlier quoted context omitted.

Hey jiggawatts - TPC is the official way to audit benchmarks in the database industry. They’ve been around for a bit, but let me know if you want more info, I’m happy to share more about them.

It sounds fundamentally busted if a competitor can submit benchmarks for someone else. TPC is great in general, but I didn't realize it had such a gaping flaw. TPC submissions take real time/$/energy/expertise, so I don't know anyone who has ever done it casually. Ex: It was a multi-company effort for the RAPIDS community to get enough API coverage & edge case optimization for an end-to-end GPU submission on the big…

Check my reply, Leo.

Re: Snowflake’s response to Databricks’ TPC-DS post

#22

Post by lmeyerov »

rxin wrote: post #21
lmeyerov wrote: post #10

Earlier quoted context omitted.

It sounds fundamentally busted if a competitor can submit benchmarks for someone else. TPC is great in general, but I didn't realize it had such a gaping flaw. TPC submissions take real time/$/energy/expertise, so I don't know anyone who has ever done it casually. Ex: It was a multi-company effort for the RAPIDS community to get enough API coverage & edge case optimization for an end-to-end GPU submission on the big…

Check my reply, Leo.

The audit question is on Databricks marketing unaudited Snowflake TPC numbers. I do think Snowflake is big enough to run TPC, but how you guys choose to market is on you.

But: I think it's cool both companies got it to $200-300. Way better than years ago. Next stop: GPUs :)

Re: Snowflake’s response to Databricks’ TPC-DS post

#23

Post by [deleted] »

rxin wrote: post #21
lmeyerov wrote: post #10

Earlier quoted context omitted.

It sounds fundamentally busted if a competitor can submit benchmarks for someone else. TPC is great in general, but I didn't realize it had such a gaping flaw. TPC submissions take real time/$/energy/expertise, so I don't know anyone who has ever done it casually. Ex: It was a multi-company effort for the RAPIDS community to get enough API coverage & edge case optimization for an end-to-end GPU submission on the big…

Check my reply, Leo.

[deleted]

Re: Snowflake’s response to Databricks’ TPC-DS post

#24

Post by bjornsing »

> At the end of the script, the overall elapsed time and the geometric mean for all the queries is computed directly by querying the history view of all TPC-DS statements that have executed on the warehouse.

The geometric mean? Really? Feels a lot easier to think in terms of arithmetic mean, and perhaps percentiles.

Re: Snowflake’s response to Databricks’ TPC-DS post

#25

Post by rxin »

lmeyerov wrote: post #22
rxin wrote: post #21

Earlier quoted context omitted.

Check my reply, Leo.

The audit question is on Databricks marketing unaudited Snowflake TPC numbers. I do think Snowflake is big enough to run TPC, but how you guys choose to market is on you. But: I think it's cool both companies got it to $200-300. Way better than years ago. Next stop: GPUs :)

Ah ok. Wasn't clear. I think some repro scripts will be available soon.

Re: Snowflake’s response to Databricks’ TPC-DS post

#26

Post by rxin »

bjornsing wrote: post #24

> At the end of the script, the overall elapsed time and the geometric mean for all the queries is computed directly by querying the history view of all TPC-DS statements that have executed on the warehouse. The geometric mean? Really? Feels a lot easier to think in terms of arithmetic mean, and perhaps percentiles.

Geometric mean is commonly used in benchmarks when the workloads consists of queries that have large (often orders of magnitude) differences in runtime.

Consider 4 queries. Two run for 1sec, and the other two 1000sec. If we look at arithmetic mean, then we are really only taking into account the large queries. But improving geometric mean would require improving all queries.

Note that I'm on the opposite side (Databricks cofounder here), so when I say that Snowflake didn't make a mistake here, you should trust me :)

Re: Snowflake’s response to Databricks’ TPC-DS post

#28

Post by rdeboo »

hiyer wrote: post #13

Performance is only one part of the story. The major advantage Snowflake (and to some extent Presto/Trino) brings to the table is it's pretty much plug and play. Spark OTOH usually requires a lot of tweaking to work reliably for your workloads.

I think the comparison was Snowflake vs Databricks SQL. Databricks SQL is a PaaS service just like Snowflake. Also, it uses their Photon engine, which is a proprietary engine written in C++. It is not Spark.

Re: Snowflake’s response to Databricks’ TPC-DS post

#29

Post by michaelhartm »

* Databricks is unethical

* Nobody should benchmark anymore, just focus on customers instead

* But hey, we just did some benchmarks and we look better than what Databricks claims

* Btw, please sign up and do some benchmarks on Snowflake, we actually ship TPC-DS dataset with Snowflake

* Btw, we agree with Databricks, let's remove the DeWitt clause, vendors should be able to benchmark each other!

* Consistency is more important than anything else!!!

Re: Snowflake’s response to Databricks’ TPC-DS post

#30

Post by hiyer »

EdwardDiego wrote: post #20
hiyer wrote: post #13

Performance is only one part of the story. The major advantage Snowflake (and to some extent Presto/Trino) brings to the table is it's pretty much plug and play. Spark OTOH usually requires a lot of tweaking to work reliably for your workloads.

Very much true. I saw a joke tweet recently something along the lines of - It's amazing how many data engineering scaling issues these days are being solved by just paying Snowflake more money. Spark does take a lot of tuning, but then I'm guessing Databricks offer that service as part of your licensing fee? (I'd hope so if they're selling a product based on FOSS code, there has to be a value add to justify it)

> I'd hope so if they're selling a product based on FOSS code, there has to be a value add to justify it

They have some proprietary features like DBIO [1]. They also have some cloud-specific features like storage autoscaling [2] that would not be available in OSS Spark. Even Delta Lake [3] used to be proprietary, but I suspect the rise of open-source frameworks like Iceberg led them to open-source it.

Shameless plug - when working at a since-shutdown competitor to Databricks, I'd come up with storage autoscaling long before them [4], so it's not unlikely that they were "inspired" by us :-) .

1. https://docs.databricks.com/spark/latest/spark-sql/dbio-comm...

2. https://databricks.com/blog/2017/12/01/transparent-autoscali...

3. https://delta.io/

4. https://www.qubole.com/blog/auto-scaling-in-qubole-with-aws-...

Post reply on HN