Databricks response to Snowflake's accusation of lacking integrity • Hacker News Board

Re: Databricks response to Snowflake's accusation of lacking integrity

#61

Post by matt123456789 » Mon, Nov 15, 2021, 9:52 PM UTC

dreyfan wrote: post #36

kartoonhero wrote: post #30

Earlier quoted context omitted.

Databricks is way more than hadoop or spark. A great analogy - Spark is a great engine but you need to design and build all of the other subsystems. Databricks is an F1 car - everything is built out. You get in and drive - FAST.

Databricks is a shit platform that encourages terrible data practices and accretion of technical debt.

As with other offerings in this space, the key to managing technical debt is to get functions out of notebooks ASAP, stage intermediate results where appropriate, and turn everything into jobs.

Re: Databricks response to Snowflake's accusation of lacking integrity

#62

Post by glogla » Mon, Nov 15, 2021, 9:58 PM UTC

redwood wrote: post #19

As much as I love seeing competition in the space and am enjoying my popcorn, I really don't understand what Databricks is doing here: this feels like a childish foodfight rather than an obsession with the customer...

Yes, the tone of those blogposts, the likelihood of fake benchmarks submitted on someone else's behalf and especially the deluge of new accounts supporting them makes me want to trust Databricks even less than the PoC my company ran with them last year and spending time with their terrible, terrible salespeople.

EDIT: I forgot lying about how open they are when all their interesting technologies (like the new sql engine and the good parts of delta) are proprietary.

Re: Databricks response to Snowflake's accusation of lacking integrity

#63

Post by bpaneural » Mon, Nov 15, 2021, 9:59 PM UTC

ignoramous wrote: post #55

scapecast wrote: post #41

The irony here is that what Databricks is doing to Snowflake is exactly what Snowflake did to AWS and Redshift. Same playbook - show that you’re better in a key metric that’s easy to understand (performance) to get the attention, but then pitch the paradigm change. In Snowflake’s case, that was separation of storage and compute. In Databrick’s case, it’s the Lakehouse Architecture. I think the reason why Snowflake is…

> I think the reason why Snowflake is so nervous because they know they can’t win this game. Isn't Databricks' delta.io, which their Data Lakehouse product builds on top of, open source? Snowflake could take the best parts from and run with it?

They could in principle. GCP, for instance, does do that. So does HP. And Databricks don't mind that as they have a strong open source legacy. But that takes away the proprietary lock-in strategy of Snowflake.

Re: Databricks response to Snowflake's accusation of lacking integrity

#64

Post by tshanmu » Mon, Nov 15, 2021, 10:02 PM UTC

drej wrote: post #43

What I find hilarious is that companies argue who can query 100 TB faster and try to sell this to people. I've been on the receiving end of offers by both of the companies in question and used both platforms (and sadly migrated some data jobs to them). While they can crunch large datasets, they are laughably slow for the datasets most people have. So while I did propose we use these solutions for our big-ish data pro…

Resume driven development FTW!

Re: Databricks response to Snowflake's accusation of lacking integrity

#65

Post by bpaneural » Mon, Nov 15, 2021, 10:06 PM UTC

strongbond wrote: post #58

kartoonhero wrote: post #48

Earlier quoted context omitted.

Please read up on Lakehouse. Data Lake + Merge support + DW performance is now possible. That is the game changer.

Do you work for Databricks?

They must do. But if you've been in this area for long enough, I'd put my money on Databricks, if anything, because of their open source integrity

Re: Databricks response to Snowflake's accusation of lacking integrity

#66

Post by laserlight » Mon, Nov 15, 2021, 10:09 PM UTC

initplus wrote: post #29

1cvmask wrote: post #7

This reminds me of the old performance ads of Oracle where they would show you how everything ran better on Oracle. They used to put those ads at airports, business lounges and the back cover of newspapers and magazines read by non-technical executives like the FT and Economist. Everyone technical knew they would game every environment to come out with superior results. I suppose it worked. As the top executives buy…

A key part of the Oracle strategy is making it a breach of license to publish any benchmarking data. No performance data about Oracle's database is allowed to be published without their approval, which means no negative results are published.

Here's some background for those who are interested [0].

[0] A solution to DeWitt clauses. https://danluu.com/anon-benchmark/

Re: Databricks response to Snowflake's accusation of lacking integrity

#67

Post by djbusby » Mon, Nov 15, 2021, 10:10 PM UTC

bloodyplonker22 wrote: post #56

Databricks is trying to punch up at the market leader. Every decent marketer knows that you should never do the opposite and punch down.

I'm crap at marketing and know the only-punch-up rule.

Re: Databricks response to Snowflake's accusation of lacking integrity

#68

Post by AdamProut » Mon, Nov 15, 2021, 10:12 PM UTC

I would say that TPC-DS and TPC-H are really table stakes benchmarks for data warehouses at this point in time (maybe they weren't 10 years ago). How to build a database that does well on them is well documented in the literature now[1][2][3][4] (maybe a few other papers). Its not easy to build such a database, but its "just" hard work and many companies have the $$ necessary to do that work. There isn't any magic or technical moat in the results for databricks (or snowflake, or redshift, etc.).

I think Databricks is overly enthusiastic about their results as they have been trying to be competitive with cloud DWs on these benchmarks for a number of years now. They have finally caught up (by building deltalake and their photon query engine which implement a number of standard DW features).

  [1] http://www.vldb.org/pvldb/vol13/p1206-dreseler.pdf
  [2] https://stratos.seas.harvard.edu/files/stratos/files/columnstoresfntdbs.pdf
  [3] https://web.stanford.edu/class/cs245/readings/c- store.pdf
  [4] http://sites.computer.org/debull/A12mar/vectorwise.pdf

Re: Databricks response to Snowflake's accusation of lacking integrity

#69

Post by fs111 » Mon, Nov 15, 2021, 10:14 PM UTC

dreyfan wrote: post #36

kartoonhero wrote: post #30

Earlier quoted context omitted.

Databricks is way more than hadoop or spark. A great analogy - Spark is a great engine but you need to design and build all of the other subsystems. Databricks is an F1 car - everything is built out. You get in and drive - FAST.

Databricks is a shit platform that encourages terrible data practices and accretion of technical debt.

Finally somebody that has used Databricks! I can't believe all the praise I read elsewhere in the comments here. Databricks is broken in so many ways, it is beyond me how anyone can like using this.

Re: Databricks response to Snowflake's accusation of lacking integrity

#70

Post by fs111 » Mon, Nov 15, 2021, 10:16 PM UTC

kartoonhero wrote: post #30

dreyfan wrote: post #16

Databricks is a rapidly approaching IPO. Trying to justify their valuation with their overpriced in-memory hadoop.

Databricks is way more than hadoop or spark. A great analogy - Spark is a great engine but you need to design and build all of the other subsystems. Databricks is an F1 car - everything is built out. You get in and drive - FAST.

> Databricks is an F1 car - everything is built out. You get in and drive - FAST.

found the databricks employee