Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Snowflake’s response to Databricks’ TPC-DS post

snowflake.com

Post reply on HN
Print view View original

41–50 of 111 posts

Re: Snowflake’s response to Databricks’ TPC-DS post

#41

Post by ttmahdy »

jiggawatts wrote: post #33
Spivak wrote: post #9

Earlier quoted context omitted.

The results are so crazy different that either Snowflake or Databricks are wrong or outright lying.

This is my point also, and I'm being downvoted for it. If two people are in disagreement about the same facts , then one of them is either misinformed or lying. It's that simple. If the only recourse seems to be to sink to the level of mud-slinging, with no clear ability to point to the audit trail and say "this is where it all went wrong", then it calls into question the value of that auditing process. I'm personall…

The TPC audit process tends to be thorough and strict.

Possibly you missed a configuration that was included in the Full Disclosure Report or Supporting Files?

The Databricks official, audited benchmark was executed against Databricks SQL which is a PaaS service that doesn't allow special tuning btw.

Re: Snowflake’s response to Databricks’ TPC-DS post

#42

Post by redis_mlc »

maslam wrote: post #8
jiggawatts wrote: post #7

Earlier quoted context omitted.

Audited how? If you look at the Snowflake response the numbers being posted by Databricks look outright faked or otherwise false.

Hey jiggawatts - TPC is the official way to audit benchmarks in the database industry. They’ve been around for a bit, but let me know if you want more info, I’m happy to share more about them.

> TPC is the official way to audit benchmarks in the database industry.

TPC is a benchmark suite for a certain problem class. It says nothing about how the databases are configured or managed.

In case you're thick, the above is a polite way of calling you a liar.

This is why Oracle and other database vendors don't allow publishing of benchmarks. It's to protect them from incompetent or lazy authors primarily.

Source: DBA.

Re: Snowflake’s response to Databricks’ TPC-DS post

#43

Post by PostThisTooFast »

pxc wrote: post #16

Can someone ELI5 what Snowflake and Databricks are? I spent a few minutes on the Databricks website once and couldn't really penetrate the marketing jargon. There are also some technical terms I don't know at all, and when I've searched for them, the top results are all more Azure stuff. Like wtf is a datalake?

“ELI5”?

Try English next time.

Re: Snowflake’s response to Databricks’ TPC-DS post

#45

Post by jiggawatts »

ttmahdy wrote: post #41
jiggawatts wrote: post #33

Earlier quoted context omitted.

This is my point also, and I'm being downvoted for it. If two people are in disagreement about the same facts , then one of them is either misinformed or lying. It's that simple. If the only recourse seems to be to sink to the level of mud-slinging, with no clear ability to point to the audit trail and say "this is where it all went wrong", then it calls into question the value of that auditing process. I'm personall…

The TPC audit process tends to be thorough and strict. Possibly you missed a configuration that was included in the Full Disclosure Report or Supporting Files? The Databricks official, audited benchmark was executed against Databricks SQL which is a PaaS service that doesn't allow special tuning btw.

I didn’t miss it. That doesn’t make it any less misleading.

Re: Snowflake’s response to Databricks’ TPC-DS post

#46

Post by Rastonbury »

throwaway984393 wrote: post #15

"Posting benchmark results is bad because it quickly becomes a race to the wrong solution. But somebody showed us sucking on a benchmark, so here's our benchmark results showing we're better."

I'm not familiar with this realm to comment on veracity of claims but it could very well be

"Posting benchmark results is bad because it quickly becomes a race to the wrong solution. Someone misrepresented our performance in a benchmark, here are the actual results."

Re: Snowflake’s response to Databricks’ TPC-DS post

#47

Post by IanCal »

pxc wrote: post #16

Can someone ELI5 what Snowflake and Databricks are? I spent a few minutes on the Databricks website once and couldn't really penetrate the marketing jargon. There are also some technical terms I don't know at all, and when I've searched for them, the top results are all more Azure stuff. Like wtf is a datalake?

Snowflake is (amongst other things but primarily to me) SQL database as a service, designed for analytical queries over large datasets.

It separates compute and storage, so there's just a big ol' pile of data and tables, then it spins up large machines to crunch the data on demand.

Data storage is cheap and the machines are expensive per hour but running for shorter times, and with little to no ops work required it can be a cheap overall system.

Bunch of other features that are handy or vital depending on your use case (instant data sharing across accounts, for example).

I've used it to transform terabytes of JSON into nice relational tables for analysts to use with very little effort.

Hopefully that's a useful overview of what kind of thing it is and where it sits.

Re: Snowflake’s response to Databricks’ TPC-DS post

#48

Post by legerdemain »

pxc wrote: post #16

Can someone ELI5 what Snowflake and Databricks are? I spent a few minutes on the Databricks website once and couldn't really penetrate the marketing jargon. There are also some technical terms I don't know at all, and when I've searched for them, the top results are all more Azure stuff. Like wtf is a datalake?

Snowflake is a hosted database that uses SQL. Two distinctions it has is that (1) it lets users pay for data storage and compute power separately and independently and (2) it takes decisions about data indexing out of your hands.

Databricks is a vendor of hosted Spark (and is operated by the creators of Spark). Spark is software for coordinating data processing jobs on multiple machines. The jobs are written using a SQL-like API that allows fairly arbitrary transformations. Databricks also offers storage using their custom virtual cloud filesystem that exposes stored datasets as DB tables.

Both vendors also offer interactive notebook functionality (although Databricks has spent more time on theirs). They're both getting into dashboarding (I think).

Ultimately, they're both selling cloud data services, and their product offerings are gradually converging.

Re: Snowflake’s response to Databricks’ TPC-DS post

#49

Post by AtlasLion »

ttmahdy wrote: post #41
jiggawatts wrote: post #33

Earlier quoted context omitted.

This is my point also, and I'm being downvoted for it. If two people are in disagreement about the same facts , then one of them is either misinformed or lying. It's that simple. If the only recourse seems to be to sink to the level of mud-slinging, with no clear ability to point to the audit trail and say "this is where it all went wrong", then it calls into question the value of that auditing process. I'm personall…

The TPC audit process tends to be thorough and strict. Possibly you missed a configuration that was included in the Full Disclosure Report or Supporting Files? The Databricks official, audited benchmark was executed against Databricks SQL which is a PaaS service that doesn't allow special tuning btw.

That doesn't allow end users any configuration, but this doesn't apply to the company itself which can apply settings from the background on behalf of end users.

Re: Snowflake’s response to Databricks’ TPC-DS post

#50

Post by AtlasLion »

rxin wrote: post #18
blobbers wrote: post #14

This is the sort of FUD testing that gets thrown back and forth between companies of all kinds. If you're in networking, it's throughput, latency or fairness. If you're in graphics its your shaders or polygons or hashes. If you're in CPUs its your clock speed. If its cameras, it's megapixels (but nobody talks about lens or real measures of clarity) If you're in silicon it's your die size (None of that has mattered fo…

Exactly. Not sure about Netflix special, but there are experts that have dedicated their professional careers to creating fair benchmarks. Snowflake should just participate in the official TPC benchmark. Disclaimer: Databricks cofounder who authored the original blog post.

The benchmark itself is kinda useless, so I don't see why they should. If you look at tpc-h for years, you had exasol as a top dog, but in the real world that meant nothing for them.
Post reply on HN