Databricks response to Snowflake's accusation of lacking integrity • Hacker News Board

Re: Databricks response to Snowflake's accusation of lacking integrity

#161

Post by feqgmmr2 » Wed, Nov 17, 2021, 4:53 PM UTC

glogla wrote: post #148

feqgmmr2 wrote: post #112

Earlier quoted context omitted.

It is not a "small" cost. The cost is proportional to the size of the data exported. For all intents and purposes, large amounts of data are locked into Snowflake. Is it theoretically possible to export a petabyte out of SF? Sure. Do I want to spend money on it? Not really. That is what I mean by the "data doesn't come out". "Exporting" a petabyte out of Databricks is a no-op. I can already read Deltalake from other…

"Exporting PB from Snowflake" is only ever relevant if you want to move from Snowflake to something else. In that case, all other migration costs (recoding, redocumenting and especially revalidating everything, if in regulated environment) are going to make any cost of data movement irrelevant. This is just FUD.

I think it's important to understand how this kind of scenario comes up. It's unusual to want to move a whole PB at one time, and yeah in that case these other costs would come up. Problem is, the cost is more insidious than that.

Consider a scenario where data is coming in periodically, say daily, from some source, server logs, sensor data, whatever. And the user wants to train models daily on the data and they also want to do some SQL. Maybe they ingest the data directly into SF and copy it out for training, or they do it the other way round, land it in object store and the ingest into SF. This is unlikely to be a humongous amount of data, it's probably not a PB. However, this adds up, maybe for some use cases it becomes a PB in a month, maybe in a quarter, maybe it only adds up to a PB in a year.

Thing is, without a Lakehouse architecture, the user will pay to store and copy that data multiple times (at least twice) no. matter. what. They may not pay for a PB in one shot, but you can bet that eventually they'll pay multiple times to store and copy that PB.

Re: Databricks response to Snowflake's accusation of lacking integrity

#162

Post by buttaphingas » Wed, Nov 17, 2021, 8:50 PM UTC

turk- wrote: post #156

glogla wrote: post #149

Earlier quoted context omitted.

According to their documentation [1], Databricks does not have this capability even for their own engines, and definitely not for "without piping data". This is what I've personally seen few times - Databricks claiming they can do something and then it turns out they can't. Buyer beware lying salespeople and HN shills. [1]: https://docs.databricks.com/administration-guide/access-cont...

I don’t understand what capability you are saying Databricks lacks. This capability is literally the entire premise of the Data Lakehouse. With Snowflake you need to export data out/or pipe data over jdbc/odbc to an external tool. With Databricks you can use SQL for data warehousing and when you need you can work with that same data using python to train an ML model without piping data out over jdbc (using the spark…

They're still lacking things in the SQL space. For example, Databricks say they're ACID compliant, but it's only on a single-table basis. Snowflake offers multi-table ACID consistency, which is something that you would expect by default in the data warehousing world. If I'm loading, say, 10 tables in parallel, I want to be able to roll-back or commit the complete set of transactions in order to maintain data consistency. I'm sure you could work around this limitation, but it would feel like a hack, especially if you're coming from a traditional DWH world (Teradata, Netezza etc.).

Snowflake now offers Scala, Java and Python support, so it would seem their capabilities are converging even more, but both with their own strengths due to their respective histories.

Re: Databricks response to Snowflake's accusation of lacking integrity

#163

Post by doppelganger1 » Thu, Nov 18, 2021, 5:40 PM UTC

geoduck14 wrote: post #155

doppelganger1 wrote: post #132

Earlier quoted context omitted.

The larger your data, the more that indexing and maintaining them hurt you. This is why they do much better at larger datasets vs small data sets. It’s all about trade offs. To overcome this, they make use of cache and if the small data is frequently accessed, the performance is generally pretty good and acceptable for most use cases.

Did anyone else notice the surge of brand new accounts that are appearing on these discussions of Databricks with pro-Databrick opinions? If we had access to IP address of the posters, I sure would be interested in looking at correlation among them.

What about my comment above is pro-Databricks? Snowflake works the same way. So do most large scale DW insert Exadata, Netezza, etc...

Does anyone else notice people questioning common sense?

Re: Databricks response to Snowflake's accusation of lacking integrity

#164

Post by doppelganger1 » Thu, Nov 18, 2021, 5:45 PM UTC

buttaphingas wrote: post #162

turk- wrote: post #156

Earlier quoted context omitted.

I don’t understand what capability you are saying Databricks lacks. This capability is literally the entire premise of the Data Lakehouse. With Snowflake you need to export data out/or pipe data over jdbc/odbc to an external tool. With Databricks you can use SQL for data warehousing and when you need you can work with that same data using python to train an ML model without piping data out over jdbc (using the spark…

They're still lacking things in the SQL space. For example, Databricks say they're ACID compliant, but it's only on a single-table basis. Snowflake offers multi-table ACID consistency, which is something that you would expect by default in the data warehousing world. If I'm loading, say, 10 tables in parallel, I want to be able to roll-back or commit the complete set of transactions in order to maintain data consiste…

Actually, you would expect that in an OLTP world. DW's for the longest time, even Oracle, recommends you disable txn to get better performance. The logic is implemented in the ETL layer. Very rarely do you need multi-table txn in large scale DW.

Snowpark is still inferior.

Re: Databricks response to Snowflake's accusation of lacking integrity

#165

Post by saj1th » Thu, Nov 18, 2021, 8:24 PM UTC

saj1th wrote: post #154

jeltz wrote: post #142

Earlier quoted context omitted.

I have not, but I do not see why it would be much slower than direct access to the storage. Databases are quite good at streaming rows.

> I do not see why it would be much slower than direct access to the storage. Implementations of protocols like ODBC/JDBC generally implement their custom on-wire binary protocols that must be marshalled to/from the lib - and the performance would vary a lot from one implementation to another. We are seeing a lot of improvements in this space though, especially with the adoption of Arrow. There is also the question o…

> There is also the question of computing for ML.

Few reasons why Databricks platform shines here.

1) Not limited by just udfs - Extensions to improve performance, including GPU acceleration in XGBoost, distributed deep learning using HorovodRunner.

2.) End to end MLOps solution - including Feature store, Model registry & Model Serving

3.) Open approach with https://www.mlflow.org/

4.) Glass box (not blackbox) model for AutoML

Re: Databricks response to Snowflake's accusation of lacking integrity

#166

Post by hello_moto » Mon, Nov 22, 2021, 6:02 AM UTC

feqgmmr2 wrote: post #106

hello_moto wrote: post #93

Earlier quoted context omitted.

It'll take a few more years until these companies fixed all the bugs and address all the scalability issues. As of today, these companies are not good enough to take on the Data Warehouse part.

Spark has always been able to handle way larger scale than any DW.

Handle what though?

Can Spark queries 100Bn structured data performing aggregation on multiple fields (or dimension?)

Re: Databricks response to Snowflake's accusation of lacking integrity

#167

Post by hello_moto » Mon, Nov 22, 2021, 6:03 AM UTC

geoduck14 wrote: post #113

feqgmmr2 wrote: post #106

Earlier quoted context omitted.

Spark has always been able to handle way larger scale than any DW.

In my previous company, we had 63 petabytes of data in Snowflake.

That sounds great: storage problem is solved.

What about large scale read via OLAP queries (y'know, the typical measures and dimensions)

Re: Databricks response to Snowflake's accusation of lacking integrity

#168

Post by doppelganger1 » Mon, Nov 22, 2021, 3:49 PM UTC

falaki wrote: post #60

scapecast wrote: post #41

The irony here is that what Databricks is doing to Snowflake is exactly what Snowflake did to AWS and Redshift. Same playbook - show that you’re better in a key metric that’s easy to understand (performance) to get the attention, but then pitch the paradigm change. In Snowflake’s case, that was separation of storage and compute. In Databrick’s case, it’s the Lakehouse Architecture. I think the reason why Snowflake is…

To be fair Apache Spark, which started long before either company existed, was built on the assumption that compute and storage should be separate. Unlike Hadoop, Spark did not come with any storage system and could read from any source.

Hadoop was built on the notion that commodity hardware, when pooled together, can be extremely cheap and powerful. The problem is, to manage it, is a nightmare. Cloudera/HWX and others were unable to reduce the management burden and their inability to pivot to a cloud based architecture really sunk their ship.