Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Databricks acquires Neon

databricks.com

Post reply on HN
Print view View original

201–210 of 234 posts

Re: Databricks acquires Neon

#201

Post by lelandbatey »

willvarfar wrote: post #30
swalsh wrote: post #15

Earlier quoted context omitted.

Really hard disagree. Coming from hadoop, databricks is utopia. It's stable, fast, scales really well if you have massive datasets. The biggest gripe in have is how crazy expensive it is.

Spark was a really big step up from hadoop. But these days just use trino or whatever. There are lots of new ways to work on data that are all bigger steps up - ergonomically, performance and price - over spark as spark was over hadoop.

I really can't speak highly enough of Trino (though I used it as AWS Athena, and this was back when Trino was called Presto). It's impressive how well it took "ever growing pile of CSV/JSON/Excel/Parquet/whatever" and let you query it via SQL as-is without transforming it and putting it into some other system.

What an impressive feat of engineering.

Re: Databricks acquires Neon

#202

Post by felixrydberg »

refset wrote: post #177
amazingamazing wrote: post #116

Earlier quoted context omitted.

I get that, but my understanding is that they opened sourced this itself, no?

It's not clear to me that the _entire_ Neon stack is OSS and available to self-host (though they do share a lot of OSS code, which is great), and in any case, it's not currently supported/documented beyond some "local development" instructions, e.g. "We do not officially support use of autoscaling externally" [0] > Can’t you use a cloud provider and have them host this for you? If it really is all OSS, then I guess t…

There are some sections of the code that is close-source. Like the Console, but the devs are very active in their Discord. There is a small community trying to self-host, and from what I have seen gotten most of it working. There isn't an easy way to start it up for a more junior developer, but if you want to self-host it you are able to since all "components" are available

Re: Databricks acquires Neon

#203

Post by steve_adams_86 »

tomjakubowski wrote: post #126
isignal wrote: post #102

Earlier quoted context omitted.

Aren't the alternatives you mentioned - icerberg and duckdb - both storage solutions while spark is a way to express distributed compute? I'm a bit out of touch with this space, is there a newer way to express distributed compute?

DuckDB is not only a storage solution. It can directly query a variety of file formats at rest, without having to re-store anything. That's one of its selling points: you can query across archival/log data stored in S3 (or wherever) without needing to "ingest" anything or double-pay to duplicate the data you've already stored.

I’m just getting into DuckDB lately and finding this feature so exciting. It’s a totally new paradigm. Such a great tool for scientists, and probably many other people. I wish I took it seriously sooner.

Re: Databricks acquires Neon

#204

Post by __turbobrew__ »

lars_francke wrote: post #93
jamesblonde wrote: post #74

Data warehousing is quickly becoming a commodity through open-source. I know a company who had 2PBs+ of data in Cloudera. But instead of moving to the cloud (and Databricks), they saved 5X costs by building their own analytics platform with Iceberg, Trino and Superset. The k8s operators are enterprise quality now. On-premises S3 is good, too. You can have great hardware (servers with 128 cpus and 1 TB) and networking…

Anyone looking for an open-source Cloudera alternative based on Kubernetes operators. We're building one (~5 years old now): https://stackable.tech/ & https://github.com/stackabletech/ On-premise open-source S3 is a problem though. MinIO is not something we're touching and other than that it looks a bit empty with enterprise ready solutions.

> On-premise open-source S3 is a problem though

Rook/ceph with object storage is pretty bulletproof: https://www.rook.io/docs/rook/v1.17/Storage-Configuration/Ob...

I do wish more systems had high quality operators out there. A lot of operators I have looked into are half baked, not reliable, or not supported.

Re: Databricks acquires Neon

#205

Post by teetertater »

apwell23 wrote: post #38
flanked-evergl wrote: post #3

Maybe unrelated but Databricks is the most annoying garbage I have ever had to use. It fascinates me how anyone uses it by choice.

Is hosting spark really that groundbreaking ? Also isn't spark kind of too complicated for 90% of enterprisey data-processing . I really don't understand the valuation for this company. Why is it so high.

Yes, spark is too complicated for most cases;

But if you're inclined to use it, databricks' setup of spark just saves you an incredible amount of time that you'd normally waste on configuration and wiring infrastructure (storage, compute, pipelines, unified access, VPNs etc). It's expensive and opinionated, but the data engineers you need to deal with spark OOM errors constantly is greater. Also databricks' default configs give you MUCH better performance out of the box than anything DIY and you don't have to fiddle with partitions and super niche config options to get even medium workloads stable

Re: Databricks acquires Neon

#206

Post by richardw »

jeffbee wrote: post #188
richardw wrote: post #154

Earlier quoted context omitted.

Not commoditising for enterprise. My last gig wouldn’t allow open source software or any company that might not be there in a decade, or which kept data anywhere but our own tenant. We’d look for the “call us” pricing rather than hate it, which I normally do. We added databricks and it was considered one of my top three achievements, because they don’t have to think about data platforms again, just focus on using it.…

> My last gig wouldn’t allow open source software or any company that might not be there in a decade I bet they had VMware all over the place.

Yup. And OpenShift. And Red Hat for Linux. And SAP. And IBM. But you know what? Tiny group of people relative to the impact, revenue and competitors. If we needed skills we clicked “buy”, 100 consultants would arrive who are experts, sort it and we’d move on. Not scratching around looking for people who know what we use and needing to learn 50 different open source tools. Coming from a much looser universe I learned to appreciate the principles for that context.

Re: Databricks acquires Neon

#207

Post by data_marsupial »

jamesblonde wrote: post #74

Data warehousing is quickly becoming a commodity through open-source. I know a company who had 2PBs+ of data in Cloudera. But instead of moving to the cloud (and Databricks), they saved 5X costs by building their own analytics platform with Iceberg, Trino and Superset. The k8s operators are enterprise quality now. On-premises S3 is good, too. You can have great hardware (servers with 128 cpus and 1 TB) and networking…

what do they use for ETL?

Re: Databricks acquires Neon

#208

Post by robertlacok »

isignal wrote: post #102
mritchie712 wrote: post #19

Earlier quoted context omitted.

Databricks started in 2013 when Spark sucked (it still does) and they aimed to make it better / faster (which they do). The product is still centered Spark, but most companies don't want or need Spark and a combination of Iceberg and DuckDB will work for 95% of companies. It's cheaper, just as fast or faster and way easier to reason about. We're building a data platform around that premise at Definite[0]. It includes…

Aren't the alternatives you mentioned - icerberg and duckdb - both storage solutions while spark is a way to express distributed compute? I'm a bit out of touch with this space, is there a newer way to express distributed compute?

I think what many people are finding out is they don’t really need distributed processing. DuckDB on a single node can get you really far, and it’s much simpler.

Re: Databricks acquires Neon

#209

Post by rajnathani »

moonikakiss wrote: post #166

This is really, really exciting. I see it as the “right” way OLTP and OLAP will converge. The OP and I built an HTAP system at SingleStore. A single database with one copy of data for both OLTP and OLAP workloads. HTAP never took off [0]. What we learned was that OLTP (Postgres) should handle OLTP, while OLAP (data warehouses/lakes) should handle OLAP, with replication between them. Designing the 'up-to-date' replica…

By OP you mean Nikita Shamgunov the founder of Neon who also founded MemSQL (SingleStore) earlier, right?

Re: Databricks acquires Neon

#210

Post by te_chris »

antruok wrote: post #163
richardw wrote: post #154

Earlier quoted context omitted.

Not commoditising for enterprise. My last gig wouldn’t allow open source software or any company that might not be there in a decade, or which kept data anywhere but our own tenant. We’d look for the “call us” pricing rather than hate it, which I normally do. We added databricks and it was considered one of my top three achievements, because they don’t have to think about data platforms again, just focus on using it.…

Enterpise also often wants a full data platform (like Databricks), not a plain data warehouse.

Exactly, look at what GCP are adding to BigQuery.
Post reply on HN