Databricks response to Snowflake's accusation of lacking integrity • Hacker News Board

Re: Databricks response to Snowflake's accusation of lacking integrity

#151

Post by glogla » Tue, Nov 16, 2021, 7:13 PM UTC

doppelganger1 wrote: post #133

initplus wrote: post #29

Earlier quoted context omitted.

A key part of the Oracle strategy is making it a breach of license to publish any benchmarking data. No performance data about Oracle's database is allowed to be published without their approval, which means no negative results are published.

Oracle Exadata is very fast but expensive. I bet it would beat a similarly sized cluster from these 2 vendors. The problem is price to performance and elasticity. Because DB and SF are in the cloud, they have a lot more options that Oracle doesn’t have. This is why Kurian left Oracle to go to Google, because LE would not allow Oracle to make cloud native products that would run in other clouds. The SF cofounders are…

Yeah, the biggest benefit something like Snowflake or Databricks or whatever AWS tools has over the more traditional technologies is the pay-as-you-go pricing.

We're are now trying to scale unnamed technology running on EC2 from 100 nodes to 200 cores and the process to buy larger license is pretty painful. If we were using Snowflake or Databricks, we could just scale it up and update our opex estimate.

Re: Databricks response to Snowflake's accusation of lacking integrity

#152

Post by saj1th » Tue, Nov 16, 2021, 7:39 PM UTC

glogla wrote: post #149

buzzscale wrote: post #144

Earlier quoted context omitted.

Thats where Databricks comes in though, you can implement row/column based security on your data on cloud object storage and use it for all your downstream use cases (Not just BI/SQL but AI/ML without piping data over JDBC/ODBC).

According to their documentation [1], Databricks does not have this capability even for their own engines, and definitely not for "without piping data". This is what I've personally seen few times - Databricks claiming they can do something and then it turns out they can't. Buyer beware lying salespeople and HN shills. [1]: https://docs.databricks.com/administration-guide/access-cont...

Check out https://databricks.com/product/unity-catalog when you get a chance. There are other solutions in this space as well.

Re: Databricks response to Snowflake's accusation of lacking integrity

#153

Post by feqgmmr2 » Tue, Nov 16, 2021, 7:42 PM UTC

glogla wrote: post #148

feqgmmr2 wrote: post #112

Earlier quoted context omitted.

It is not a "small" cost. The cost is proportional to the size of the data exported. For all intents and purposes, large amounts of data are locked into Snowflake. Is it theoretically possible to export a petabyte out of SF? Sure. Do I want to spend money on it? Not really. That is what I mean by the "data doesn't come out". "Exporting" a petabyte out of Databricks is a no-op. I can already read Deltalake from other…

"Exporting PB from Snowflake" is only ever relevant if you want to move from Snowflake to something else. In that case, all other migration costs (recoding, redocumenting and especially revalidating everything, if in regulated environment) are going to make any cost of data movement irrelevant. This is just FUD.

It's very relevant if you ever want to do serious ML or anything other than SQL. Of course Snowflake wants you to think that you never need another platform. Every customer knows that's not the case.

Re: Databricks response to Snowflake's accusation of lacking integrity

#154

Post by saj1th » Tue, Nov 16, 2021, 8:58 PM UTC

jeltz wrote: post #142

doppelganger1 wrote: post #137

Earlier quoted context omitted.

Have you ever tried to train models on large data sets over JDBC/ODBC? it’s terrible even with parallelism. Having direct access to the underlying storage and being able to bypass sucking a lot of data over a small straw is a game changer. That is one advantage that Spark and Databricks have over Snowflake.

I have not, but I do not see why it would be much slower than direct access to the storage. Databases are quite good at streaming rows.

> I do not see why it would be much slower than direct access to the storage.

Implementations of protocols like ODBC/JDBC generally implement their custom on-wire binary protocols that must be marshalled to/from the lib - and the performance would vary a lot from one implementation to another. We are seeing a lot of improvements in this space though, especially with the adoption of Arrow.

There is also the question of computing for ML. Data scientists today use several tools/frameworks ranging from scikit-learn/XGBoost to PyTorch/Keras/TensorFlow - to name a few. Enabling data scientists to use these frameworks against near-realtime data without worrying about provisioning infrastructure or managing dependencies or adding an additional export-to-cloud-storage hop is a game changer IMO.

Re: Databricks response to Snowflake's accusation of lacking integrity

#155

Post by geoduck14 » Tue, Nov 16, 2021, 9:03 PM UTC

doppelganger1 wrote: post #132

sanketsarang wrote: post #126

Earlier quoted context omitted.

I did work on making a database myself, and I must say that querying 100TB fast, let alone storing 100TB of data, is a real problem. Some companies (very few) don't have much choice but to use a DB that works on 100TB. If you do have small data, then you have a lot of options. But if your data is large, then you have very few options. So it is correct to be competing on how fast a DB can query 100TB of data; while at…

The larger your data, the more that indexing and maintaining them hurt you. This is why they do much better at larger datasets vs small data sets. It’s all about trade offs. To overcome this, they make use of cache and if the small data is frequently accessed, the performance is generally pretty good and acceptable for most use cases.

Did anyone else notice the surge of brand new accounts that are appearing on these discussions of Databricks with pro-Databrick opinions?

If we had access to IP address of the posters, I sure would be interested in looking at correlation among them.

Re: Databricks response to Snowflake's accusation of lacking integrity

#156

Post by turk- » Tue, Nov 16, 2021, 9:17 PM UTC

glogla wrote: post #149

buzzscale wrote: post #144

Earlier quoted context omitted.

Thats where Databricks comes in though, you can implement row/column based security on your data on cloud object storage and use it for all your downstream use cases (Not just BI/SQL but AI/ML without piping data over JDBC/ODBC).

According to their documentation [1], Databricks does not have this capability even for their own engines, and definitely not for "without piping data". This is what I've personally seen few times - Databricks claiming they can do something and then it turns out they can't. Buyer beware lying salespeople and HN shills. [1]: https://docs.databricks.com/administration-guide/access-cont...

I don’t understand what capability you are saying Databricks lacks. This capability is literally the entire premise of the Data Lakehouse. With Snowflake you need to export data out/or pipe data over jdbc/odbc to an external tool. With Databricks you can use SQL for data warehousing and when you need you can work with that same data using python to train an ML model without piping data out over jdbc (using the spark engine). One security model, one dataset, multiple use cases (AI/ML/BI/SQL) on one platform.

Re: Databricks response to Snowflake's accusation of lacking integrity

#157

Post by khc » Tue, Nov 16, 2021, 9:45 PM UTC

geoduck14 wrote: post #155

doppelganger1 wrote: post #132

Earlier quoted context omitted.

The larger your data, the more that indexing and maintaining them hurt you. This is why they do much better at larger datasets vs small data sets. It’s all about trade offs. To overcome this, they make use of cache and if the small data is frequently accessed, the performance is generally pretty good and acceptable for most use cases.

Did anyone else notice the surge of brand new accounts that are appearing on these discussions of Databricks with pro-Databrick opinions? If we had access to IP address of the posters, I sure would be interested in looking at correlation among them.

with most people working from home, not sure if this heuristic works.

disclaimer: works for databricks, but not on spark, and first time posting in this thread

Re: Databricks response to Snowflake's accusation of lacking integrity

#158

Post by bpaneural » Wed, Nov 17, 2021, 7:14 AM UTC

buttaphingas wrote: post #122

ignoramous wrote: post #55

Earlier quoted context omitted.

> I think the reason why Snowflake is so nervous because they know they can’t win this game. Isn't Databricks' delta.io, which their Data Lakehouse product builds on top of, open source? Snowflake could take the best parts from and run with it?

Delta is open source, but Databricks keeps optimizations for themselves as proprietary. I'm not sure why it would be any better than Snowflake's solution, which is automatically deployed across multiple AZs as a fully HA system and gives full ACID transaction compliance across any number of tables (not just per-table).

Essentially with Databricks making Delta open source, you can move away from Databricks to EMR or Presto (with their own optimizations) without incurring a data tax. You're also able to move between cloud providers at ease as data sits in low cost buckets.

Re: Databricks response to Snowflake's accusation of lacking integrity

#159

Post by buttaphingas » Wed, Nov 17, 2021, 12:51 PM UTC

turk- wrote: post #74

glogla wrote: post #57

Earlier quoted context omitted.

In what way is lakehouse architecture beneficial over something like Snowflake or BigQuery? I understand the appeal over having lake and warehouse as separate components, but with those native cloud warehouses, you can already do everything a lake does.

With a datawarehouse, you can only interface with your data in SQL. With big query and snowflake, your data is locked away in a proprietary format not accessible by other compute platforms. You need to export/copy your data to a different system to train an ML model in python or R. With the lakehouse, you can use python, R and Scala, (not just SQL) to interface with your data. You can use multiple compute engines (sp…

You can use Scala, Java and Python with Snowflake now, as well as process structured, semi-structured and unstructured data. So I guess that means it doesn't fit into the data warehouse category, but is not a lakehouse either.

Re: Databricks response to Snowflake's accusation of lacking integrity

#160

Post by buttaphingas » Wed, Nov 17, 2021, 1:11 PM UTC

turk- wrote: post #140

glogla wrote: post #130

Earlier quoted context omitted.

My experience is that's how it looks at first. But it is hard to actually make use of lake or lakehouse openness. You can access data in Snowflake or BigQuery using JDBC or Python clients. You do pay for the compute that reads the data for you. You cannot access the data in storage directly. You can access data in lakehouse directly, by going to cloud storage. That has two major challenges: Lakehouse formats aren't e…

Here is the thing with the lakehouse though, you have flexibility and don’t need to use multiple engines to achieve the lakehouse vision. Databricks has all the security features a redshift / snowflake does so you can secure databases and tables rather than s3 buckets. It does get more complex if you want to introduce multiple engines but at least you have the option to make that trade off if you want to. If you want…

But you do still have to secure the S3 buckets, right? And I guess also secure the infrastructure you have to deploy in order to run Databricks. Plus then configure for cross-AZ failover etc. So you get flexibility, but I would think at the cost of much more human labor to get it up and running.

Snowflake uses the Arrow data format with their drivers, so is plenty fast enough when retrieving data in general. But it would be way less efficient if a data scientist just does a SELECT * to bring everything back from a table to load into a notebook.

Snowflake has had Scala support since earlier in the year, along with Java UDFs, and also just announced Python support - not a Python connector, but executing Python code directly on the Snowflake platform. Not GA yet though.