Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Databricks acquires Neon

databricks.com

Post reply on HN
Print view View original

171–180 of 234 posts

Re: Databricks acquires Neon

#171

Post by winwang »

MOARDONGZPLZ wrote: post #169
winwang wrote: post #159

Earlier quoted context omitted.

What options do you use? I don't work for Databricks but I am building my own data infra startup, so I'd like to hear what "good" looks like!

Eh you don’t even need to go through all the trouble building a startup. imo Neon was interesting and filled a niche while open source solutions were really gaining maturity and adoption. Now they have, lots and lots of recommendations in this comment section, so my sense is that building a startup would be like reinventing the Neon wheel, just too late. Perhaps, depending on licensing, running OSS as a software is v…

Oh, my startup isn't about Postgres, but rather a GPU-accelerated Spark: https://news.ycombinator.com/item?id=43964505

What are some bad UX choices you generally dislike in data products?

Re: Databricks acquires Neon

#172

Post by moonikakiss »

antruok wrote: post #165
bbohan wrote: post #161

Earlier quoted context omitted.

SingleStore been doing that for years . Unistore been strugglin

I wonder why Singlestore has been so unpopular (at least I never hear about it). Quick guess is that HTAP itself isn’t a significant feature requirement, maybe just a cherry on top of other major db features.

it was a couple things. 1. It's really really hard to replace anyone's OLTP. 2. OLTP and OLAP are owned by such different teams. Who do you make your champion? 3. The modern HTAP dream is possible without something like SingleStore. You need a columnstore that can keep up with your OLTP tables and provide transactional correctness. Who cares if it's all within one system

ps: I worked at SingleStore. https://www.mooncake.dev/blog/htap-is-dead

Re: Databricks acquires Neon

#173

Post by ako »

moonikakiss wrote: post #162
ako wrote: post #146

Earlier quoted context omitted.

ETL to bring all your data into Databricks/Snowflake is a lot of effort. Much better if your OLTP data already exists in Databricks and you directly access it from your OLAP layer.

With the push towards open table formats (Iceberg) from both Snowflake and Databricks, it's even harder to get your Postgres OLTP tables ready for OLAP. The problem isn't in the CDC / replication tools in the market. The problem is that columnar stores (especially Iceberg) are not designed for the write /upserts patterns of OLTP systems. They just can't keep up... This is a big problem we're hoping to solve at Moonca…

Is Iceberg involved in every read/write? I thought it was mostly metadata?

Re: Databricks acquires Neon

#175

Post by jenny91 »

drewnick wrote: post #97
jenny91 wrote: post #58

It's my understanding that Neon had some tech to basically "wake up" the DB when a request came out -- so you could "scale down to zero," if you will. I was hoping to explore this for small personal projects: I by far prefer Postgres and would love an isolated database per project. Is there an alternative for that? Scale-to-zero postgres, basically?

For small personal projects, coolify (featured recently here on HN) lets you quickly stand up postgres with SSL, etc. and get a connection string in seconds. You can deploy in the same project or expose pg to the world like neon does. One click turns it off, or you can just leave it on. A $5 VM will run a lot of small postgres. I use both neon and coolify, and could live with either, though apples and oranges when it…

I'm not afraid of running servers, that was not the point. The point was exactly that I wanted a serverless postgres.

If I can throw together a random project, completely isolated, that costs $0.10 per month, that enables me to do many orders more random projects than something that costs me $5 per month.

Re: Databricks acquires Neon

#176

Post by lamp_book »

Nate75Sanders wrote: post #105
isignal wrote: post #102

Earlier quoted context omitted.

Aren't the alternatives you mentioned - icerberg and duckdb - both storage solutions while spark is a way to express distributed compute? I'm a bit out of touch with this space, is there a newer way to express distributed compute?

Flink. It has more momentum than Spark right now.

Flink is designed around streaming first, while Spark is built around batch first and you're likely best off selecting accordingly. Though any streaming application likely needs batch processing to some degree. Latency vs throughput.

Re: Databricks acquires Neon

#177

Post by refset »

amazingamazing wrote: post #116
refset wrote: post #103

Earlier quoted context omitted.

> you use S3 as bottomless storage for Postgres [...] Why are people paying? It's vastly more complicated to do this efficiently than you might imagine. Postgres' internal architecture is built around a very different set of assumptions (pages, WAL, local disk etc.) than what the S3 API offers.

I get that, but my understanding is that they opened sourced this itself, no?

It's not clear to me that the _entire_ Neon stack is OSS and available to self-host (though they do share a lot of OSS code, which is great), and in any case, it's not currently supported/documented beyond some "local development" instructions, e.g. "We do not officially support use of autoscaling externally" [0]

> Can’t you use a cloud provider and have them host this for you?

If it really is all OSS, then I guess the moat is the impressive execution of this team.

[0] https://github.com/neondatabase/autoscaling

Re: Databricks acquires Neon

#178

Post by t0mas88 »

swyx wrote: post #153
jamesblonde wrote: post #74

Data warehousing is quickly becoming a commodity through open-source. I know a company who had 2PBs+ of data in Cloudera. But instead of moving to the cloud (and Databricks), they saved 5X costs by building their own analytics platform with Iceberg, Trino and Superset. The k8s operators are enterprise quality now. On-premises S3 is good, too. You can have great hardware (servers with 128 cpus and 1 TB) and networking…

if Databricks just wanted a row DB they couldve done postgres themselves. paying this much for Neon i think is a sign that Neon has something special they want (which, knowing their marketing line, is "independently scalable storage and compute for postgres")

That sounds like AWS Aurora?

Re: Databricks acquires Neon

#179

Post by tristan957 »

refset wrote: post #177
amazingamazing wrote: post #116

Earlier quoted context omitted.

I get that, but my understanding is that they opened sourced this itself, no?

It's not clear to me that the _entire_ Neon stack is OSS and available to self-host (though they do share a lot of OSS code, which is great), and in any case, it's not currently supported/documented beyond some "local development" instructions, e.g. "We do not officially support use of autoscaling externally" [0] > Can’t you use a cloud provider and have them host this for you? If it really is all OSS, then I guess t…

The only component which is not currently open source is the control plane.

Re: Databricks acquires Neon

#180

Post by yencabulator »

swyx wrote: post #153
jamesblonde wrote: post #74

Data warehousing is quickly becoming a commodity through open-source. I know a company who had 2PBs+ of data in Cloudera. But instead of moving to the cloud (and Databricks), they saved 5X costs by building their own analytics platform with Iceberg, Trino and Superset. The k8s operators are enterprise quality now. On-premises S3 is good, too. You can have great hardware (servers with 128 cpus and 1 TB) and networking…

if Databricks just wanted a row DB they couldve done postgres themselves. paying this much for Neon i think is a sign that Neon has something special they want (which, knowing their marketing line, is "independently scalable storage and compute for postgres")

Easy quick cheap forks of database state for AI agents to muck with.
Post reply on HN