Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Databricks to Buy Data-Management Startup Tabular

wsj.com

Post reply on HN
Print view View original

11–20 of 23 posts

Re: Databricks to Buy Data-Management Startup Tabular

#11

Post by datadrivenangel »

jonnat wrote: post #9

So, will Databricks deprioritize Delta Lake in favor or Iceberg, or will they try to derail Iceberg development now they they got the team that originally built it at Netflix? Edit: from the Tabular CEO announcement Databricks reached out to me and proposed a collaboration that could bring Iceberg and Delta closer together [...] I’m excited to have the opportunity to work with Databricks and the broader Delta communi…

So Delta will eventually become better but not before Databricks gets a year or three of advantage over iceberg and open source delta.

Re: Databricks to Buy Data-Management Startup Tabular

#12

Post by indoordin0saur »

jetru wrote: post #5
Pils wrote: post #3

Seems bad for Snowflake? Iceberg is a big part of Snowflake's data lake offering, and I assumed it was a Snowflake-originated OSS project until this announcement (all Snowflake products have snow related names).

Iceberg is not Snowflake originated, it was built by folks at Netflix - the same folks who built Tabular. But yes, this is definitely bad for Snowflake, Databricks can position itself as a very strong competitor with this move and moving more towards Iceberg.

So this makes it sound like a positive thing for Iceberg as a format. Others seem to be suggesting that DataBricks will be working to undermine the format in favor of their older Delta Lake format but that seems overly cynical.

Re: Databricks to Buy Data-Management Startup Tabular

#13

Post by indoordin0saur »

jonnat wrote: post #9

So, will Databricks deprioritize Delta Lake in favor or Iceberg, or will they try to derail Iceberg development now they they got the team that originally built it at Netflix? Edit: from the Tabular CEO announcement Databricks reached out to me and proposed a collaboration that could bring Iceberg and Delta closer together [...] I’m excited to have the opportunity to work with Databricks and the broader Delta communi…

Doesn't it sound like they'll try and move the two formats closer together so that there isn't such a format war? IDK how it would benefit Databricks to ruin either format if they're now such huge stakeholders in them both.

Either way, I just want to know which format to pick. I've been chief data engineer at my current company for about a year and would like to be able to move off of plain parquet files in my lake but I'm not sure what table format to choose.

Re: Databricks to Buy Data-Management Startup Tabular

#14

Post by jamesblonde »

I was at MySQL when Oracle bought lowly Innobase (3 people), but it was the open-source tech that MySQL built on. MySQL subsequently tried to build their own transaction engine (Falcon), but failed. This deal feels like Oracle = Databricks, Snowflake = MySQL

The big difference - innodb got like 3m, Tabular 1bn!

Re: Databricks to Buy Data-Management Startup Tabular

#15

Post by jetru »

indoordin0saur wrote: post #12
jetru wrote: post #5

Earlier quoted context omitted.

Iceberg is not Snowflake originated, it was built by folks at Netflix - the same folks who built Tabular. But yes, this is definitely bad for Snowflake, Databricks can position itself as a very strong competitor with this move and moving more towards Iceberg.

So this makes it sound like a positive thing for Iceberg as a format. Others seem to be suggesting that DataBricks will be working to undermine the format in favor of their older Delta Lake format but that seems overly cynical.

Yea, in theory, the way its stated, it would be a plus for the industry and Iceberg consolidation.

How it will actually play out, who knows?

Re: Databricks to Buy Data-Management Startup Tabular

#16

Post by indoordin0saur »

jetru wrote: post #15
indoordin0saur wrote: post #12

Earlier quoted context omitted.

So this makes it sound like a positive thing for Iceberg as a format. Others seem to be suggesting that DataBricks will be working to undermine the format in favor of their older Delta Lake format but that seems overly cynical.

Yea, in theory, the way its stated, it would be a plus for the industry and Iceberg consolidation. How it will actually play out, who knows?

My guess is Data Bricks saw the popularity of Iceberg and realized that they were starting to look a little irrelevant still trying to promote their competing Delta Lake format. Have an Iceberg lakehouse? Well then Databricks just didn't seem very relevant to you. With their purchase of Tabular they're given some legitimacy when they start marketing their products as being iceberg compatible. This doesn't signal to me that Iceberg is going to be harmed in the near or medium term.

Re: Databricks to Buy Data-Management Startup Tabular

#17

Post by artwr »

Sounds like a really good move by Databricks, in particular because a lot of the main platforms had implementations of catalogs to the Iceberg Spec, and several vendors, Snowflake included was starting to support Iceberg as an external Table format.

I have similar questions about the future of Delta Lake, but not really about the future of Iceberg, that's what the Apache Foundation is for after all. There are enough large enterprise players relying on this (Apple, Netflix, ...) to keep the project going for a while.

Re: Databricks to Buy Data-Management Startup Tabular

#19

Post by waterlx »

indoordin0saur wrote: post #13
jonnat wrote: post #9

So, will Databricks deprioritize Delta Lake in favor or Iceberg, or will they try to derail Iceberg development now they they got the team that originally built it at Netflix? Edit: from the Tabular CEO announcement Databricks reached out to me and proposed a collaboration that could bring Iceberg and Delta closer together [...] I’m excited to have the opportunity to work with Databricks and the broader Delta communi…

Doesn't it sound like they'll try and move the two formats closer together so that there isn't such a format war? IDK how it would benefit Databricks to ruin either format if they're now such huge stakeholders in them both. Either way, I just want to know which format to pick. I've been chief data engineer at my current company for about a year and would like to be able to move off of plain parquet files in my lake b…

Hi, in case you did not find the answer yet. In my hamble opinion: - choose Iceberg: If you have several computing/query engines other than Spark, like Presto, Flink. Iceberg has a great extraction and design for a engine-independent table format. But its learning cost is relative high - choose Delta: If you only have Spark and would like to be deeply binded with Databricks - choose Hudi: If you would like to use data lake out-of-the-box and it is quite easy to use. - If your data is updated frequently, like streaming, check https://paimon.apache.org/ if you would like to be deeply binded with Flink

Re: Databricks to Buy Data-Management Startup Tabular

#20

Post by indoordin0saur »

waterlx wrote: post #19
indoordin0saur wrote: post #13

Earlier quoted context omitted.

Doesn't it sound like they'll try and move the two formats closer together so that there isn't such a format war? IDK how it would benefit Databricks to ruin either format if they're now such huge stakeholders in them both. Either way, I just want to know which format to pick. I've been chief data engineer at my current company for about a year and would like to be able to move off of plain parquet files in my lake b…

Hi, in case you did not find the answer yet. In my hamble opinion: - choose Iceberg: If you have several computing/query engines other than Spark, like Presto, Flink. Iceberg has a great extraction and design for a engine-independent table format. But its learning cost is relative high - choose Delta: If you only have Spark and would like to be deeply binded with Databricks - choose Hudi: If you would like to use dat…

Thank you! Sounds like iceberg is the best then. I'm very allergic to lock-in. Currently we're very Spark heavy and our query engine is AWS Redshift Serverless. The recent AWS Glue Catalog support for Iceberg seems to make this promising.
Post reply on HN