Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Databricks response to Snowflake's accusation of lacking integrity

databricks.com

Post reply on HN
Print view View original

101–110 of 168 posts

Re: Databricks response to Snowflake's accusation of lacking integrity

#101

Post by drawturkey »

glogla wrote: post #46
kartoonhero wrote: post #30

Earlier quoted context omitted.

Databricks is way more than hadoop or spark. A great analogy - Spark is a great engine but you need to design and build all of the other subsystems. Databricks is an F1 car - everything is built out. You get in and drive - FAST.

> Databricks is an F1 car F1 cars really unreliable and need a lot of engineers to keep running, are very expensive, and completely impractical in normal use. They are fast but only on very specific roads, they couldn't survive on normal roads. What do you know, you might be right! :D

You nailed it. Meanwhile the rest of the world just needs a camry.

Re: Databricks response to Snowflake's accusation of lacking integrity

#102

Post by thrtlvlmidnight »

drawturkey wrote: post #92
michaelhartm wrote: post #2

Data Wars: Snowflake vs Databricks (0 - 2)?

Snowflake has way more revenue, is worth 3 times more than Databricks and is growing faster. I'd say Snowflake is still in the lead. Plus, just look at Snowflake's customer list. It's a "who's who", Databricks is a "Who's that?".

I took a look at Databricks public customer case studies[1] and haven't a clue who any of these companies are:

Atlassian? Adobe? ExxonMobil? PagerDuty? McAfee? HSBC? Starbucks? AstraZeneca? GlaxoSmithKline? Comcast? FINRA? Regeneron? Riot Games? Nielsen? HP? Conde Nast? Viacom? McGraw-Hill? Cisco? NBCUniversal?

Hopefully they can scale to the enterprise soon.

[1]https://databricks.com/customers

Re: Databricks response to Snowflake's accusation of lacking integrity

#103

Post by nojvek »

benjaminwootton wrote: post #53

Ive been following this and it’s kind of embarrassing to watch. I love working with Databricks and Snowflake. They both knock it out of the park for their respective use case. They’re amazing products. It makes no sense to fall out about this though. For a 100TB dataset with a funky calculation, Spark will trounce Snowflake. For a 1 row dataset, Snowflake will return before the spark job has been serialised.

Why would Spark trounce Snowflake. What makes it inherently so much faster at 100TB jobs?

Also what kind of queries are we talking about?

Re: Databricks response to Snowflake's accusation of lacking integrity

#104

Post by syntaxfree »

kartoonhero wrote: post #37
avip wrote: post #6

I've used both products in production. Both are good++. The blog wars seem extremely ridiculous to me. I don't recall ever choosing one over another based on how fast it runs on some imaginary arbitrary dataset.

Its not ridiculous at all. This is the coming of age for a brand new data architecture. One of the biggest FUDs for a data lake architecture is performance - and this benchmark should put that concern to rest.

I don’t know, “coming of age” seems to imply that there’s some pre-maturity period out of which something is emerging.

Re: Databricks response to Snowflake's accusation of lacking integrity

#105

Post by feqgmmr2 »

drawturkey wrote: post #98
buttaphingas wrote: post #84

Earlier quoted context omitted.

Databricks isn't open source, as they keep hold of all the IP that makes it much better than OS Spark. Whether you buy Snowflake or Databricks, you're buying proprietary software.

This is very true. They make the lowest common denominator parts "open source" but control all of the commits. Also the query engine used for this benchmark is proprietary, closed source (Photon)

The 'open' here refers to the data. Delta lake can be read/written by multiple open source engines, not just Spark. Not to mention, if you want you can use Databricks with Parquet, though the experience won't be as good.

But with Snowflake, the data never comes out. Can't use Spark/Trino/Flink... on data in SF.

Re: Databricks response to Snowflake's accusation of lacking integrity

#106

Post by feqgmmr2 »

hello_moto wrote: post #93
kartoonhero wrote: post #48

Earlier quoted context omitted.

Please read up on Lakehouse. Data Lake + Merge support + DW performance is now possible. That is the game changer.

It'll take a few more years until these companies fixed all the bugs and address all the scalability issues. As of today, these companies are not good enough to take on the Data Warehouse part.

Spark has always been able to handle way larger scale than any DW.

Re: Databricks response to Snowflake's accusation of lacking integrity

#107

Post by drawturkey »

feqgmmr2 wrote: post #105
drawturkey wrote: post #98

Earlier quoted context omitted.

This is very true. They make the lowest common denominator parts "open source" but control all of the commits. Also the query engine used for this benchmark is proprietary, closed source (Photon)

The 'open' here refers to the data. Delta lake can be read/written by multiple open source engines, not just Spark. Not to mention, if you want you can use Databricks with Parquet, though the experience won't be as good. But with Snowflake, the data never comes out. Can't use Spark/Trino/Flink... on data in SF.

Not entirely true. There is a bi-directional Spark connector for Snowflake written by Databricks. And exporting your data in bulk out of Snowflake into any number of open formats is incredibly easy using the COPY INTO command. You can also use Snowflake on top of Parquet and even Delta Lake.

This is the problem. Both Snowflake and Databricks are spreading FUD and otherwise smart people are falling for it.

Re: Databricks response to Snowflake's accusation of lacking integrity

#108

Post by StephenJGL »

drej wrote: post #43

What I find hilarious is that companies argue who can query 100 TB faster and try to sell this to people. I've been on the receiving end of offers by both of the companies in question and used both platforms (and sadly migrated some data jobs to them). While they can crunch large datasets, they are laughably slow for the datasets most people have. So while I did propose we use these solutions for our big-ish data pro…

Very true. You have to understand the actual capabilities and your actual requirements. We work with petabyte size datasets and BigQuery is hard to beat. Our other reporting systems are still all in MySQL though.

Re: Databricks response to Snowflake's accusation of lacking integrity

#109

Post by [deleted] »

feqgmmr2 wrote: post #105
drawturkey wrote: post #98

Earlier quoted context omitted.

This is very true. They make the lowest common denominator parts "open source" but control all of the commits. Also the query engine used for this benchmark is proprietary, closed source (Photon)

The 'open' here refers to the data. Delta lake can be read/written by multiple open source engines, not just Spark. Not to mention, if you want you can use Databricks with Parquet, though the experience won't be as good. But with Snowflake, the data never comes out. Can't use Spark/Trino/Flink... on data in SF.

[deleted]

Re: Databricks response to Snowflake's accusation of lacking integrity

#110

Post by drawturkey »

feqgmmr2 wrote: post #105
drawturkey wrote: post #98

Earlier quoted context omitted.

This is very true. They make the lowest common denominator parts "open source" but control all of the commits. Also the query engine used for this benchmark is proprietary, closed source (Photon)

The 'open' here refers to the data. Delta lake can be read/written by multiple open source engines, not just Spark. Not to mention, if you want you can use Databricks with Parquet, though the experience won't be as good. But with Snowflake, the data never comes out. Can't use Spark/Trino/Flink... on data in SF.

Agreed there is a small cost, but it is possible, which is at odds with your statement "with Snowflake, the data never comes out".
Post reply on HN