Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Databricks acquires Neon

databricks.com

Post reply on HN
Print view View original

161–170 of 234 posts

Re: Databricks acquires Neon

#161

Post by bbohan »

antruok wrote: post #145
datadrivenangel wrote: post #131

Earlier quoted context omitted.

But why would you buy an operational DB from Databricks? The only thing that makes sense is Databricks flailing to maintain market cap.

In addition to the AI use cases, sometimes you wanna share the data warehouse data in oltp way for fast lookups and high concurrency. Not sure whether Neon will do that but I hope so. One example from Snowflake is hybrid tables which adds rowstore next to columnar. OLAP + OLTP = HTAP

SingleStore been doing that for years . Unistore been strugglin

Re: Databricks acquires Neon

#162

Post by moonikakiss »

ako wrote: post #146
datadrivenangel wrote: post #131

Earlier quoted context omitted.

But why would you buy an operational DB from Databricks? The only thing that makes sense is Databricks flailing to maintain market cap.

ETL to bring all your data into Databricks/Snowflake is a lot of effort. Much better if your OLTP data already exists in Databricks and you directly access it from your OLAP layer.

With the push towards open table formats (Iceberg) from both Snowflake and Databricks, it's even harder to get your Postgres OLTP tables ready for OLAP.

The problem isn't in the CDC / replication tools in the market.

The problem is that columnar stores (especially Iceberg) are not designed for the write /upserts patterns of OLTP systems.

They just can't keep up...

This is a big problem we're hoping to solve at Mooncake [0]. Turn Iceberg into an operational columnstore. So that it can be keep up (https://www.mooncake.dev/

Re: Databricks acquires Neon

#163

Post by antruok »

richardw wrote: post #154
jamesblonde wrote: post #74

Data warehousing is quickly becoming a commodity through open-source. I know a company who had 2PBs+ of data in Cloudera. But instead of moving to the cloud (and Databricks), they saved 5X costs by building their own analytics platform with Iceberg, Trino and Superset. The k8s operators are enterprise quality now. On-premises S3 is good, too. You can have great hardware (servers with 128 cpus and 1 TB) and networking…

Not commoditising for enterprise. My last gig wouldn’t allow open source software or any company that might not be there in a decade, or which kept data anywhere but our own tenant. We’d look for the “call us” pricing rather than hate it, which I normally do. We added databricks and it was considered one of my top three achievements, because they don’t have to think about data platforms again, just focus on using it.…

Enterpise also often wants a full data platform (like Databricks), not a plain data warehouse.

Re: Databricks acquires Neon

#164

Post by pjdbruin »

lars_francke wrote: post #93
jamesblonde wrote: post #74

Data warehousing is quickly becoming a commodity through open-source. I know a company who had 2PBs+ of data in Cloudera. But instead of moving to the cloud (and Databricks), they saved 5X costs by building their own analytics platform with Iceberg, Trino and Superset. The k8s operators are enterprise quality now. On-premises S3 is good, too. You can have great hardware (servers with 128 cpus and 1 TB) and networking…

Anyone looking for an open-source Cloudera alternative based on Kubernetes operators. We're building one (~5 years old now): https://stackable.tech/ & https://github.com/stackabletech/ On-premise open-source S3 is a problem though. MinIO is not something we're touching and other than that it looks a bit empty with enterprise ready solutions.

Great to see cost-effective alternatives to Cloudera and Databricks! We’ve spent three years building IOMETE, a self-hosted data lakehouse that combines Apache Iceberg and Spark, designed to run natively on Kubernetes. We’re focused on on-premises deployments to address the growing need for data sovereignty and low TCO, with a streamlined setup for large-scale analytics. Early adopters are seeing strong results. Curious about your experience with Trino and Superset—any tips for optimizing performance at scale?

Re: Databricks acquires Neon

#165

Post by antruok »

bbohan wrote: post #161
antruok wrote: post #145

Earlier quoted context omitted.

In addition to the AI use cases, sometimes you wanna share the data warehouse data in oltp way for fast lookups and high concurrency. Not sure whether Neon will do that but I hope so. One example from Snowflake is hybrid tables which adds rowstore next to columnar. OLAP + OLTP = HTAP

SingleStore been doing that for years . Unistore been strugglin

I wonder why Singlestore has been so unpopular (at least I never hear about it). Quick guess is that HTAP itself isn’t a significant feature requirement, maybe just a cherry on top of other major db features.

Re: Databricks acquires Neon

#166

Post by moonikakiss »

This is really, really exciting. I see it as the “right” way OLTP and OLAP will converge.

The OP and I built an HTAP system at SingleStore. A single database with one copy of data for both OLTP and OLAP workloads. HTAP never took off [0].

What we learned was that OLTP (Postgres) should handle OLTP, while OLAP (data warehouses/lakes) should handle OLAP, with replication between them.

Designing the 'up-to-date' replication between these systems is hard.... columnar stores just aren’t built for OLTP‑style writes, and can't keep up with your OLTP tables.

Let’s see if Databricks and Neon can pull this off

“give me up‑to‑date Postgres tables in Unity Catalog", no debezium --> kafka --> flink --> Iceberg. With Spark jobs in the back ensuring that Iceberg is an optimal state.

https://www.mooncake.dev/blog/htap-is-dead

Re: Databricks acquires Neon

#168

Post by orangechairs »

antruok wrote: post #165
bbohan wrote: post #161

Earlier quoted context omitted.

SingleStore been doing that for years . Unistore been strugglin

I wonder why Singlestore has been so unpopular (at least I never hear about it). Quick guess is that HTAP itself isn’t a significant feature requirement, maybe just a cherry on top of other major db features.

SingleStore (nee MemSQL) was/is niche -- great fit inside banks in particular.

Re: Databricks acquires Neon

#169

Post by MOARDONGZPLZ »

winwang wrote: post #159
MOARDONGZPLZ wrote: post #94

Earlier quoted context omitted.

Databricks is the Jira of dealing with data. No one wants to use it, it sucks, there are too many features to try to appease all possible users but none of them particularly good, and there are substantially better options now than there were not long ago. I would never, ever use it by choice.

What options do you use? I don't work for Databricks but I am building my own data infra startup, so I'd like to hear what "good" looks like!

Eh you don’t even need to go through all the trouble building a startup. imo Neon was interesting and filled a niche while open source solutions were really gaining maturity and adoption. Now they have, lots and lots of recommendations in this comment section, so my sense is that building a startup would be like reinventing the Neon wheel, just too late. Perhaps, depending on licensing, running OSS as a software is viable.

Re: Databricks acquires Neon

#170

Post by dustingetz »

swyx wrote: post #153
jamesblonde wrote: post #74

Data warehousing is quickly becoming a commodity through open-source. I know a company who had 2PBs+ of data in Cloudera. But instead of moving to the cloud (and Databricks), they saved 5X costs by building their own analytics platform with Iceberg, Trino and Superset. The k8s operators are enterprise quality now. On-premises S3 is good, too. You can have great hardware (servers with 128 cpus and 1 TB) and networking…

if Databricks just wanted a row DB they couldve done postgres themselves. paying this much for Neon i think is a sign that Neon has something special they want (which, knowing their marketing line, is "independently scalable storage and compute for postgres")

"Time is the denominator"
Post reply on HN