Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Databricks response to Snowflake's accusation of lacking integrity

databricks.com

Post reply on HN
Print view View original

141–150 of 168 posts

Re: Databricks response to Snowflake's accusation of lacking integrity

#141

Post by jeltz »

autokad wrote: post #89
drej wrote: post #43

What I find hilarious is that companies argue who can query 100 TB faster and try to sell this to people. I've been on the receiving end of offers by both of the companies in question and used both platforms (and sadly migrated some data jobs to them). While they can crunch large datasets, they are laughably slow for the datasets most people have. So while I did propose we use these solutions for our big-ish data pro…

its my experience if its just 10s of GBs then use 'normal' solutions. if TB then spark is great for that. note I have only used DataBricks & Spark, no snowflake.

PostgreSQL and MySQL can handle a few TB just fine. It is when you reach over 10TB that you need something else.

Re: Databricks response to Snowflake's accusation of lacking integrity

#142

Post by jeltz »

doppelganger1 wrote: post #137
glogla wrote: post #130

Earlier quoted context omitted.

My experience is that's how it looks at first. But it is hard to actually make use of lake or lakehouse openness. You can access data in Snowflake or BigQuery using JDBC or Python clients. You do pay for the compute that reads the data for you. You cannot access the data in storage directly. You can access data in lakehouse directly, by going to cloud storage. That has two major challenges: Lakehouse formats aren't e…

Have you ever tried to train models on large data sets over JDBC/ODBC? it’s terrible even with parallelism. Having direct access to the underlying storage and being able to bypass sucking a lot of data over a small straw is a game changer. That is one advantage that Spark and Databricks have over Snowflake.

I have not, but I do not see why it would be much slower than direct access to the storage. Databases are quite good at streaming rows.

Re: Databricks response to Snowflake's accusation of lacking integrity

#143

Post by glogla »

doppelganger1 wrote: post #137
glogla wrote: post #130

Earlier quoted context omitted.

My experience is that's how it looks at first. But it is hard to actually make use of lake or lakehouse openness. You can access data in Snowflake or BigQuery using JDBC or Python clients. You do pay for the compute that reads the data for you. You cannot access the data in storage directly. You can access data in lakehouse directly, by going to cloud storage. That has two major challenges: Lakehouse formats aren't e…

Have you ever tried to train models on large data sets over JDBC/ODBC? it’s terrible even with parallelism. Having direct access to the underlying storage and being able to bypass sucking a lot of data over a small straw is a game changer. That is one advantage that Spark and Databricks have over Snowflake.

Have you tried to implement row- and column-based security on direct access to cloud storage? It flat out does not work.

Sadly, those things are mutually exclusive at the moment and with the way things are deployed here (large multi-tenant platforms), the security has to take priority.

But if that's not your situation, then obviously it makes sense to make use of that!

Re: Databricks response to Snowflake's accusation of lacking integrity

#144

Post by buzzscale »

glogla wrote: post #143
doppelganger1 wrote: post #137

Earlier quoted context omitted.

Have you ever tried to train models on large data sets over JDBC/ODBC? it’s terrible even with parallelism. Having direct access to the underlying storage and being able to bypass sucking a lot of data over a small straw is a game changer. That is one advantage that Spark and Databricks have over Snowflake.

Have you tried to implement row- and column-based security on direct access to cloud storage? It flat out does not work. Sadly, those things are mutually exclusive at the moment and with the way things are deployed here (large multi-tenant platforms), the security has to take priority. But if that's not your situation, then obviously it makes sense to make use of that!

Thats where Databricks comes in though, you can implement row/column based security on your data on cloud object storage and use it for all your downstream use cases (Not just BI/SQL but AI/ML without piping data over JDBC/ODBC).

Re: Databricks response to Snowflake's accusation of lacking integrity

#145

Post by feqgmmr2 »

glogla wrote: post #143
doppelganger1 wrote: post #137

Earlier quoted context omitted.

Have you ever tried to train models on large data sets over JDBC/ODBC? it’s terrible even with parallelism. Having direct access to the underlying storage and being able to bypass sucking a lot of data over a small straw is a game changer. That is one advantage that Spark and Databricks have over Snowflake.

Have you tried to implement row- and column-based security on direct access to cloud storage? It flat out does not work. Sadly, those things are mutually exclusive at the moment and with the way things are deployed here (large multi-tenant platforms), the security has to take priority. But if that's not your situation, then obviously it makes sense to make use of that!

This is exactly what FAANGs do with their data platforms. There are literally hundreds of groups within these companies with very strict data isolation requirements between them. Pretty sure something like that is either already possible or will be very soon, there's just too much prior art here.

Re: Databricks response to Snowflake's accusation of lacking integrity

#146

Post by imslowbutnice »

benjaminwootton wrote: post #53

Ive been following this and it’s kind of embarrassing to watch. I love working with Databricks and Snowflake. They both knock it out of the park for their respective use case. They’re amazing products. It makes no sense to fall out about this though. For a 100TB dataset with a funky calculation, Spark will trounce Snowflake. For a 1 row dataset, Snowflake will return before the spark job has been serialised.

What are you talking about. Spark isn't even used, and TPC DS is not a funky calculation at all. It's supposed to be a collection of typical datawarehouse type queries. Although I'm not really sure what funky means, but why would Spark trounce Snowflake on "funky" calculation at all. Do you mean an ML algorithm, and are you implying that TPC-DS has anything close to an ML Algorithm? And why would Snowflake perform better on returning one row, they are columnar stored.

Re: Databricks response to Snowflake's accusation of lacking integrity

#147

Post by saj1th »

glogla wrote: post #143
doppelganger1 wrote: post #137

Earlier quoted context omitted.

Have you ever tried to train models on large data sets over JDBC/ODBC? it’s terrible even with parallelism. Having direct access to the underlying storage and being able to bypass sucking a lot of data over a small straw is a game changer. That is one advantage that Spark and Databricks have over Snowflake.

Have you tried to implement row- and column-based security on direct access to cloud storage? It flat out does not work. Sadly, those things are mutually exclusive at the moment and with the way things are deployed here (large multi-tenant platforms), the security has to take priority. But if that's not your situation, then obviously it makes sense to make use of that!

> Have you tried to implement row- and column-based security on direct access to cloud storage? It flat out does not work.

It is a solved problem. Essentially you need a central place ( with decentralized ownership for the datamesh fans ) to specify the ACLS ( row-based, column-based, attribute-based etc.) - and an enforcement layer that understands these ACLs. There are many solutions, including the ones from Databricks. Data discovery, lineage, data quality etc., go hand in glove.

Security is front and centre for almost all organizations now.

Re: Databricks response to Snowflake's accusation of lacking integrity

#148

Post by glogla »

feqgmmr2 wrote: post #112
drawturkey wrote: post #107

Earlier quoted context omitted.

Not entirely true. There is a bi-directional Spark connector for Snowflake written by Databricks. And exporting your data in bulk out of Snowflake into any number of open formats is incredibly easy using the COPY INTO command. You can also use Snowflake on top of Parquet and even Delta Lake. This is the problem. Both Snowflake and Databricks are spreading FUD and otherwise smart people are falling for it.

It is not a "small" cost. The cost is proportional to the size of the data exported. For all intents and purposes, large amounts of data are locked into Snowflake. Is it theoretically possible to export a petabyte out of SF? Sure. Do I want to spend money on it? Not really. That is what I mean by the "data doesn't come out". "Exporting" a petabyte out of Databricks is a no-op. I can already read Deltalake from other…

"Exporting PB from Snowflake" is only ever relevant if you want to move from Snowflake to something else. In that case, all other migration costs (recoding, redocumenting and especially revalidating everything, if in regulated environment) are going to make any cost of data movement irrelevant.

This is just FUD.

Re: Databricks response to Snowflake's accusation of lacking integrity

#149

Post by glogla »

buzzscale wrote: post #144
glogla wrote: post #143

Earlier quoted context omitted.

Have you tried to implement row- and column-based security on direct access to cloud storage? It flat out does not work. Sadly, those things are mutually exclusive at the moment and with the way things are deployed here (large multi-tenant platforms), the security has to take priority. But if that's not your situation, then obviously it makes sense to make use of that!

Thats where Databricks comes in though, you can implement row/column based security on your data on cloud object storage and use it for all your downstream use cases (Not just BI/SQL but AI/ML without piping data over JDBC/ODBC).

According to their documentation [1], Databricks does not have this capability even for their own engines, and definitely not for "without piping data".

This is what I've personally seen few times - Databricks claiming they can do something and then it turns out they can't. Buyer beware lying salespeople and HN shills.

[1]: https://docs.databricks.com/administration-guide/access-cont...

Re: Databricks response to Snowflake's accusation of lacking integrity

#150

Post by glogla »

thrtlvlmidnight wrote: post #102
drawturkey wrote: post #92

Earlier quoted context omitted.

Snowflake has way more revenue, is worth 3 times more than Databricks and is growing faster. I'd say Snowflake is still in the lead. Plus, just look at Snowflake's customer list. It's a "who's who", Databricks is a "Who's that?".

I took a look at Databricks public customer case studies[1] and haven't a clue who any of these companies are: Atlassian? Adobe? ExxonMobil? PagerDuty? McAfee? HSBC? Starbucks? AstraZeneca? GlaxoSmithKline? Comcast? FINRA? Regeneron? Riot Games? Nielsen? HP? Conde Nast? Viacom? McGraw-Hill? Cisco? NBCUniversal? Hopefully they can scale to the enterprise soon. [1] https://databricks.com/customers

Lists of "references" like these are worthless. Because larger companies tend to be fragmented, especially companies that have more complicated business lines and are used to departments and divisions acting independently.

You know what, our company uses both Snowflake and Databricks.

For Databricks, there's one or two projects that someone built on it running in production. For Snowflake, there's a sizeable use because we bought a smaller company that used it for reporting and warehousing. Neither of them are "the chosen tool" and will see any growth unless wind changes. But we could be (F50 company) counted as reference by both I guess.

Post reply on HN