Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Snowflake’s response to Databricks’ TPC-DS post

snowflake.com

Post reply on HN
Print view View original

91–100 of 111 posts

Re: Snowflake’s response to Databricks’ TPC-DS post

#91

Post by 62951413 »

geoduck14 wrote: post #58
kevindeasis wrote: post #35

Earlier quoted context omitted.

They are a data warehouse with analytics? So data warehouse as a service in the cloud? So they can collect data from different places like sql, images, etc. I think a better question would be what type of data can't they ingest? Once you have your data i guess you can run some analytics to find out what your data tells you

I'd like to add some points: Ive used Snowflake for several years. Snowflake works with structured and semi-structured data (think spreadsheets and JSON). I've never tried working with pics or videos - and I'm not sure it would make sense to do that. I've evaluated Databricks. It works with the above mentioned structured and semi-structured data. I also suspect it could process unstructured data. My understanding is…

Databricks used to be an Apache Spark as a service company. And Spark is a predominantly Scala code base. PySpark is just a Python binding for the real engine popular in ML circles. In the last couple of years the Databricks platform migrated from open-source Spark to a new proprietary engine written in C++.

Re: Snowflake’s response to Databricks’ TPC-DS post

#93

Post by [deleted] »

AtlasLion wrote: post #50
rxin wrote: post #18

Earlier quoted context omitted.

Exactly. Not sure about Netflix special, but there are experts that have dedicated their professional careers to creating fair benchmarks. Snowflake should just participate in the official TPC benchmark. Disclaimer: Databricks cofounder who authored the original blog post.

The benchmark itself is kinda useless, so I don't see why they should. If you look at tpc-h for years, you had exasol as a top dog, but in the real world that meant nothing for them.

[deleted]

Re: Snowflake’s response to Databricks’ TPC-DS post

#94

Post by bjornsing »

ttmahdy wrote: post #87
bjornsing wrote: post #56

Earlier quoted context omitted.

> But improving geometric mean would require improving all queries. No. Improving the geometric mean only requires reducing the product of their execution times. So if you can make the two 1 ms queries execute in 0.5 ms at the expense of the two 1000 ms queries taking 1800 ms each then that’s an improvement in terms of geometric mean. So… kind of QED. The geometric mean is not easy to reason about.

Usually making a 1 ms query execute in 0.5 ms is a lot harder than making a 10 second query execute in 5 second. One of the benefits of geometric mean is that all queries have "equal" weight in the metric, this keeps vendors from focusing on the long running queries and ignoring the short running ones. It is one way to balance between long and short query performance. A similar concept is applied to TPC-DS for data l…

> Usually making a 1 ms query execute in 0.5 ms is a lot harder than making a 10 second query execute in 5 second.

Eh, okay... It produces the same reduction in geometric mean though, right?

Re: Snowflake’s response to Databricks’ TPC-DS post

#95

Post by buttaphingas »

tyingq wrote: post #57
aptxkid wrote: post #2

Personally I think it’s a great response and very well written. I didn’t jump on the congrats-Databricks wagon when the result first came out because of the weird front page comparison against snowflake. Both companies are doing great work. Focusing on building a better product for your customer is much more meaningful than making your competitor look bad.

It is well written, but there's some sleight of hand here and there too. Like using your lowest tier product to demonstrate price/performance against a competitor's highest tier. The Snowflake lowest tier doesn't have failover, for example...or compliance features.

This is incorrect. Every edition of Snowflake is deployed across multiple availability zones with automatic failover in the case of failure or AZ outage. This is included in the price and requires no configuration by the customer. Cross-cloud/region failover requires the top edition and a few lines of SQL to configure (plus cloud egress costs for data replication).

The higher editions of Snowflake include features like materialised views, dynamic data masking, BYOK, PCI & HIPAA compliance etc., non of which are required for the benchmark.

Re: Snowflake’s response to Databricks’ TPC-DS post

#96

Post by Tanjreeve »

glogla wrote: post #79
socaldata wrote: post #65

Take all the problems you have had with data warehousing and throw them in a proprietary cloud. That is Snowflake. They are the best today. Databricks started with the cloud datalake, sitting natively on parquet and using cloud native tools, fully open. Recently they added SQL to help democratize the data in the data lake versus moving it back and forth into a proprietary data warehouse. The selling point in Databric…

Delta lake is not meaningfully more "open" than whatever Snowflake (or BigQuery and Redshift) are doing. It does not require any less "moving data around" With all these, the data sits on cloud storage and compute is done by cloud machines - the difference between Databricks and the others is that with Databricks, you can take a look at that bucket. But you're not going to be able to do much with that data without pa…

Delta lake is very much open. You can install delta lake and run it yourself. It's a transaction layer running over parquet files. You can go to the delta.io GitHub and install binaries yourself. Snowflake cannot be run independently of their cloud.

The rest of this is some vague claims of Databricks being unreliable techbros blah blah which is just emotionally charged hot air rather than being based on anything.

RE who to pick. Run them side by side. Use snowflake for non technical staff/BI load in prepared cuts of data. it's batteries included and less knobs to twiddle for optimisation. Databricks/spark has a learning code and isn't suitable for non-technical staff. But it gives a lot more options for processing for all the stuff that doesn't fit neatly into data clustering.

Re: Snowflake’s response to Databricks’ TPC-DS post

#97

Post by tyingq »

buttaphingas wrote: post #95
tyingq wrote: post #57

Earlier quoted context omitted.

It is well written, but there's some sleight of hand here and there too. Like using your lowest tier product to demonstrate price/performance against a competitor's highest tier. The Snowflake lowest tier doesn't have failover, for example...or compliance features.

This is incorrect. Every edition of Snowflake is deployed across multiple availability zones with automatic failover in the case of failure or AZ outage. This is included in the price and requires no configuration by the customer. Cross-cloud/region failover requires the top edition and a few lines of SQL to configure (plus cloud egress costs for data replication). The higher editions of Snowflake include features li…

I'm getting it from Snowflake's own page:

https://www.snowflake.com/pricing/

Amongst other things, listed under the enterprise tier, and not lower tiers, is "Database failover and failback for business continuity".

"The higher editions of Snowflake include features like materialised views, dynamic data masking, BYOK, PCI & HIPAA compliance etc., non of which are required for the benchmark."

Yeah, but they are referencing a price/performance comparison to a Databricks tier that DOES have those things. That's the point. Update your own numbers with a lower tier, but don't update the competitor tier too?

Re: Snowflake’s response to Databricks’ TPC-DS post

#98

Post by buttaphingas »

kthejoker2 wrote: post #82

Snowflake conceding they have a 700% markup between Standard and Premium editons which has zero impact on query performance is ... well, it's something. I'd start squeezing my sales engineers about that, definitely not sustainable... Also proof that lakehouse and spot compute price performance economics are here to stay, that's good for customers. Otherwise, as a vendor blog post with nothing but self-reported perfor…

I've used Snowflake for the past few years, and it's worth pointing out that when it comes to overall cost, there's a lot you get with Snowflake for free. For example, they have HA across 3 AZs out of the box, included in the price and with no configuration required.

If I'm reading what Databricks published correctly, it seems that they've only used 1 driver node for this benchmark, in other words it's a dev setup. If they want to compare apples-to-apples then they should configure, and price, a multi-AZ HA set-up.

I'm not sure if this is still applicable to Photon, however - can anyone confirm?

Re: Snowflake’s response to Databricks’ TPC-DS post

#99

Post by buttaphingas »

tyingq wrote: post #97
buttaphingas wrote: post #95

Earlier quoted context omitted.

This is incorrect. Every edition of Snowflake is deployed across multiple availability zones with automatic failover in the case of failure or AZ outage. This is included in the price and requires no configuration by the customer. Cross-cloud/region failover requires the top edition and a few lines of SQL to configure (plus cloud egress costs for data replication). The higher editions of Snowflake include features li…

I'm getting it from Snowflake's own page: https://www.snowflake.com/pricing/ Amongst other things, listed under the enterprise tier, and not lower tiers, is "Database failover and failback for business continuity". "The higher editions of Snowflake include features like materialised views, dynamic data masking, BYOK, PCI & HIPAA compliance etc., non of which are required for the benchmark." Yeah, but they are referen…

The "failover and failback for business continuity" is specifically for cross-region/cloud, i.e. this is something you explicitly have to do. tbh I've never used it, as I guess this would be only for very large accounts. But all editions have automatic failover between AZs out-of-the-box.

[Edit] Highly Available would be a better description per region, as that's out of the box with no configuration. e.g. if a node dies, your cluster will automatically heal and resubmit your query. If there's an entire AZ outage, your query should be resubmitted in another AZ. I think this is why failover/back is called out separately, as that's not automatic, incurs additional costs etc. Here's a link with an explanation: www.snowflake.com/blog/how-to-make-data-protection-and-high-availability-for-analytics-fast-and-easy

I didn't know DB did MVs, masking etc., so yes, that makes sense. Maybe a better idea would be to have a minimum offering comparison, and then a maximum offering comparison (with multi-AZ failover, masking feature costs etc. included) - the reality for a customer would be somewhere between those extremes.

Re: Snowflake’s response to Databricks’ TPC-DS post

#100

Post by gibneyMI »

rxin wrote: post #17
jiggawatts wrote: post #7

Earlier quoted context omitted.

Audited how? If you look at the Snowflake response the numbers being posted by Databricks look outright faked or otherwise false.

There's an official TPC process to audit and review the benchmark process. This debate can be easiest settled by everybody participating in the official benchmark, like we (Databricks) did. The official review process is significantly more complicated than just offering a static dataset that's been highly optimized for answering the exact set of queries. It includes data loading, data maintenance (insert and delete d…

Credit to you for these amazing benchmark scores via an official process. You've certainly proved to naysayers such as Stonebreaker that lakes and warehouses can be combined in a performant manner!

Shame on your for quoting a fake non-official score for Snowflake in your blog post with crude suggestions to make it seem you're showing an apples-to-apples comparison.

I run a BI org in an F500 company that uses both Databricks & Snowflake on AWS. I can tell you that such dishonest shenanigans take away much from your truly noteworthy technical achievements and make me not want to buy your stuff for lack of integrity. Not very long ago, Azure+GigaOM did a similar blog post with fake numbers on AWS Redshift and it resulted in my department and a bunch of large F500 enterprises that I know moving away from Synapse for lack of integrity.

On many occasions, I've felt that Databricks product management and sales teams lack integrity (especially the folks from Uber & VMW) and such moves only amplify this impression. Your sales guys use arm-twisting tactics to meet quotas and your PM execs. are clueless about your technology and industry. My suggestion is to overhaul some of these teams and cull the rot - it is taking away from the great work your engineers and Berkley research teams are doing.

Post reply on HN