Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Databricks response to Snowflake's accusation of lacking integrity

databricks.com

Post reply on HN
Print view View original

131–140 of 168 posts

Re: Databricks response to Snowflake's accusation of lacking integrity

#131

Post by glogla »

solidangle wrote: post #77
glogla wrote: post #33

Earlier quoted context omitted.

Clickhouse is good if you're building application. It has lot of great features and incredible performance, but there's an expectancy that people using it know what they're doing and can work around its limitations (like limited support for joins and sql in general). Something like Snowflake works much better when you're building a platform that you can give to two hundred data analysts or various skills spread over…

When was the last time you used Databricks? You should definitely try it again. Their product offering has improved a lot in the past few years. > broad feature set My experience is that the feature sets of Snowflake and Databricks are very similar. Both have time travel support. Snowflake has materialized views, but Databricks has Delta Live Tables. Databricks has a distributed Pandas API, but Snowflake recently int…

Snowflake has much more advanced data security - table, column, and row level security and dynamic data masking policies. The zero-copy cloning is also pretty useful for CI/CD (pretty much the one practical way to do blue-green deployment for data application).

Databricks has some interesting features (we were originally interested in it as "nice UI" for our AWS data lake for citizen data scientists - using it for industrialized processing was price impractical compared to AWS Glue) but the security seems lacking - it goes just table level and only in SQL and Spark, with R you can't have security at all.

I really liked the Databricks UI and integrated visualizations, though, that's where they are better than Snowflake I think. Of course, they gained those by buying open source Redash.io and ending it.

The part that ended our PoC with them was when they gave us a price quote for expected number of users, the management was like "ok that sounds reasonable" until I told them that's just license and does not include EC2 costs - the real cost would be at least twice. That made everyone angry.

Re: Databricks response to Snowflake's accusation of lacking integrity

#132

Post by doppelganger1 »

sanketsarang wrote: post #126
drej wrote: post #43

What I find hilarious is that companies argue who can query 100 TB faster and try to sell this to people. I've been on the receiving end of offers by both of the companies in question and used both platforms (and sadly migrated some data jobs to them). While they can crunch large datasets, they are laughably slow for the datasets most people have. So while I did propose we use these solutions for our big-ish data pro…

I did work on making a database myself, and I must say that querying 100TB fast, let alone storing 100TB of data, is a real problem. Some companies (very few) don't have much choice but to use a DB that works on 100TB. If you do have small data, then you have a lot of options. But if your data is large, then you have very few options. So it is correct to be competing on how fast a DB can query 100TB of data; while at…

The larger your data, the more that indexing and maintaining them hurt you. This is why they do much better at larger datasets vs small data sets. It’s all about trade offs.

To overcome this, they make use of cache and if the small data is frequently accessed, the performance is generally pretty good and acceptable for most use cases.

Re: Databricks response to Snowflake's accusation of lacking integrity

#133

Post by doppelganger1 »

initplus wrote: post #29
1cvmask wrote: post #7

This reminds me of the old performance ads of Oracle where they would show you how everything ran better on Oracle. They used to put those ads at airports, business lounges and the back cover of newspapers and magazines read by non-technical executives like the FT and Economist. Everyone technical knew they would game every environment to come out with superior results. I suppose it worked. As the top executives buy…

A key part of the Oracle strategy is making it a breach of license to publish any benchmarking data. No performance data about Oracle's database is allowed to be published without their approval, which means no negative results are published.

Oracle Exadata is very fast but expensive. I bet it would beat a similarly sized cluster from these 2 vendors. The problem is price to performance and elasticity. Because DB and SF are in the cloud, they have a lot more options that Oracle doesn’t have. This is why Kurian left Oracle to go to Google, because LE would not allow Oracle to make cloud native products that would run in other clouds. The SF cofounders are ex Oracle engineers and LE was not interested in creating a cloud native DB from scratch. If he did, we wouldn’t have a SF computing right now.

Re: Databricks response to Snowflake's accusation of lacking integrity

#134

Post by doppelganger1 »

belter wrote: post #75
1cvmask wrote: post #7

This reminds me of the old performance ads of Oracle where they would show you how everything ran better on Oracle. They used to put those ads at airports, business lounges and the back cover of newspapers and magazines read by non-technical executives like the FT and Economist. Everyone technical knew they would game every environment to come out with superior results. I suppose it worked. As the top executives buy…

Who could forget the Unbreakable and Unhackable Campaign... The "Unbreakable" Marketing Campaign: https://www.oreilly.com/library/view/the-oracle-hackers/9780... https://www.zdnet.com/article/invincible-oracle-not-so-secur...

The first thing unbreakable Linux did was break.

Re: Databricks response to Snowflake's accusation of lacking integrity

#135

Post by doppelganger1 »

scapecast wrote: post #41

The irony here is that what Databricks is doing to Snowflake is exactly what Snowflake did to AWS and Redshift. Same playbook - show that you’re better in a key metric that’s easy to understand (performance) to get the attention, but then pitch the paradigm change. In Snowflake’s case, that was separation of storage and compute. In Databrick’s case, it’s the Lakehouse Architecture. I think the reason why Snowflake is…

SF spreads a lot of FUD saying that DB can’t perform, and it was true. DB then went out and hired a lot of engineering talent with a diverse background and has been investing a lot of money in being a best in class SQL offering, so what do you do? You do something to get people’s attention. They’re saying, “hey, we have great performance too, you should also look at us for your SQL workloads.”

Re: Databricks response to Snowflake's accusation of lacking integrity

#136

Post by doppelganger1 »

glogla wrote: post #57
scapecast wrote: post #41

The irony here is that what Databricks is doing to Snowflake is exactly what Snowflake did to AWS and Redshift. Same playbook - show that you’re better in a key metric that’s easy to understand (performance) to get the attention, but then pitch the paradigm change. In Snowflake’s case, that was separation of storage and compute. In Databrick’s case, it’s the Lakehouse Architecture. I think the reason why Snowflake is…

In what way is lakehouse architecture beneficial over something like Snowflake or BigQuery? I understand the appeal over having lake and warehouse as separate components, but with those native cloud warehouses, you can already do everything a lake does.

Big Query&Data Proc, Redshift&EMR, Synapse&HDR are tied to the cloud vendors. You can’t move easily from AWS stack to GCP without refactoring. Switching costs are higher.

Snowflake and Databricks are multicloud. The different is that Snowflake is more like a SaaS solution and only does SQL. Databricks is more than just SQL. It has all the data science, machine learning information, built into it. Snowflake has Snowpark but it’s every limited and so you are more likely to have to buy more products to build out your capabilities and integrate them with Snowflake. With Databricks it is more out of the box in terms of capabilities. Databricks also runs in your cloud account which has trade offs. It can be harder to get going and more complex but you end up with a lot more flexibility and you own your data and have complete control over it. While Snowflake gives you control of your data with their tools, everything has to go through Snowflake and incur their tax to get to it. You pay for simplicity, which many customers are ok with because they see value in it. On the contrary, a lot of customers see value in having more control and options. This market is big enough for everyone - it’s really just about market share.

Re: Databricks response to Snowflake's accusation of lacking integrity

#137

Post by doppelganger1 »

glogla wrote: post #130
turk- wrote: post #74

Earlier quoted context omitted.

With a datawarehouse, you can only interface with your data in SQL. With big query and snowflake, your data is locked away in a proprietary format not accessible by other compute platforms. You need to export/copy your data to a different system to train an ML model in python or R. With the lakehouse, you can use python, R and Scala, (not just SQL) to interface with your data. You can use multiple compute engines (sp…

My experience is that's how it looks at first. But it is hard to actually make use of lake or lakehouse openness. You can access data in Snowflake or BigQuery using JDBC or Python clients. You do pay for the compute that reads the data for you. You cannot access the data in storage directly. You can access data in lakehouse directly, by going to cloud storage. That has two major challenges: Lakehouse formats aren't e…

Have you ever tried to train models on large data sets over JDBC/ODBC? it’s terrible even with parallelism. Having direct access to the underlying storage and being able to bypass sucking a lot of data over a small straw is a game changer. That is one advantage that Spark and Databricks have over Snowflake.

Re: Databricks response to Snowflake's accusation of lacking integrity

#138

Post by doppelganger1 »

inetknght wrote: post #86

Snowflake accuses other companies of lacking integrity? I really wish I could block all of Snowflake's domain from my inbox. Sadly, Google encourages spammers to just create a new email address. So I get a few emails each month from Snowflake who ask me to try their products. I've never done business with them and there's no unsubscribe link. Fuck Snowflake for thinking it has any room to talk about integrity.

What I find comical is they accuse Databricks of lacking integrity but they don’t actually call out anything except their benchmark was faster than what Databricks did in Snowflake. Databricks then reruns the benchmark and says the only reason that Snowflake’s was faster was because of the built in dataset they used. Databricks was able to match Snowflakes numbers using it but when they loaded the actual data set, it was much slower, which is how a proper TPC benchmark is supposed to happen. They then said that Databricks blog doesn’t match the TPC results, but when I looked at them, they do match. I guess Snowflake just expects people to take arguments at face value. Then I saw someone on LinkedIn complaining that Databricks must have used some beta version. I didn’t see a beta version being used, but that kind of goes out the window when Databricks follows up and then posts that they matched Snowflake when they used their built in TPC data set.

This is funny and interesting to watch but also a distraction I feel. Amazon says it best when they say, “Leaders start with the customer and work backwards. They work vigorously to earn and keep customer trust. Although leaders pay attention to competitors, they obsess over customers.”

Re: Databricks response to Snowflake's accusation of lacking integrity

#139

Post by doppelganger1 »

1cvmask wrote: post #35
initplus wrote: post #29

Earlier quoted context omitted.

A key part of the Oracle strategy is making it a breach of license to publish any benchmarking data. No performance data about Oracle's database is allowed to be published without their approval, which means no negative results are published.

They also sue you for so many other reasons. It's like the management hierarchy joke that Oracle is a litigious law firm with a sales team. https://palisadecompliance.com/oracle-org-chart/

Also, if a sales rep or manager are struggling to make their numbers, they will audit customers.

Re: Databricks response to Snowflake's accusation of lacking integrity

#140

Post by turk- »

glogla wrote: post #130
turk- wrote: post #74

Earlier quoted context omitted.

With a datawarehouse, you can only interface with your data in SQL. With big query and snowflake, your data is locked away in a proprietary format not accessible by other compute platforms. You need to export/copy your data to a different system to train an ML model in python or R. With the lakehouse, you can use python, R and Scala, (not just SQL) to interface with your data. You can use multiple compute engines (sp…

My experience is that's how it looks at first. But it is hard to actually make use of lake or lakehouse openness. You can access data in Snowflake or BigQuery using JDBC or Python clients. You do pay for the compute that reads the data for you. You cannot access the data in storage directly. You can access data in lakehouse directly, by going to cloud storage. That has two major challenges: Lakehouse formats aren't e…

Here is the thing with the lakehouse though, you have flexibility and don’t need to use multiple engines to achieve the lakehouse vision. Databricks has all the security features a redshift / snowflake does so you can secure databases and tables rather than s3 buckets. It does get more complex if you want to introduce multiple engines but at least you have the option to make that trade off if you want to.

If you want simplicity, you can limit your engine to Databricks. You can also use JDBC/ODBC with Databricks if you want to use other tools that don’t support the delta format/parquet but piping data over JDBC/ODBC doesn’t scale with any tool to large datasets. Databricks has all the capabilities of big query/snowflake/redshift but none of those tools support python/r/scala. Their engines need to be rewritten from the ground up in order to do so.

Post reply on HN