Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Databricks in talks to acquire startup Neon for about $1B

upstartsmedia.com

Post reply on HN
Print view View original

91–100 of 177 posts

Re: Databricks in talks to acquire startup Neon for about $1B

#91

Post by creeksai »

rogermavis wrote: post #89
creeksai wrote: post #87

Earlier quoted context omitted.

You mentioned earlier about how long it would take to acquire a new cluster in Databricks, but you are comparing it here to something that's always on here. In a much larger environment, your setup is not really practical to have a lot of people collaborating. Note that Databricks SQL Serverless these days can be provisioned in a few seconds.

> you are comparing it here to something that's always on That's the point. Our org was told databricks would solve problems we just didn't have. Serverful has some wonderful advantages: simplicity, (ironically) cheaper (than something running just 3-4 hours a day but which costs 10x), familiarity, reliability. Serverless also has advantages, but only if it runs smoothly, doesn't take an eternity to boot, isn't prohi…

I totally understand if Databricks doesn't fit your use cases.

But you are doing a disingenuous comparison here because one can keep a "serverful" cluster up without shutting it down, and in that case, you'd never need to wait for anything to boot up. If you shut down your EC2 instances, it will also take time to boot up. Alternatively, you can use the (relatively new) serverless offering from them that gets you compute resources in seconds.

Re: Databricks in talks to acquire startup Neon for about $1B

#92

Post by cactusfrog »

richardw wrote: post #50
forgetfulness wrote: post #33

What is the lowdown on Databricks? Their bread and butter were hosted Spark and notebooks. As tasks done in Spark over a data lake began to be delegated wholesale to columnar store ELT, they tried to pivot to "lake houses", then I sort of lost track of them after I got out of Spark myself. Did Delta Lake ever catch on? Where are they going now?

Capture enterprise AI enthusiasm by providing a 1-stop shop for data and AI, optionally hosted on your own cloud tenant. Keep deploying functionality so clients never need another supplier. Partner with SAP, OpenAI, anyone who holds market share. Buy anyone that either helps growth or might help a competitor grow. Enterprise view: delegate AI environment to Databricks unless you’re a real player. Market is too chaoti…

A nice UI for a data lake house is underrated. I use AWS Athena at my work and it is just so bad for no good reason. For example, big columns of text are expanded outwards making reading the subsequent columns impossible.

Re: Databricks in talks to acquire startup Neon for about $1B

#93

Post by rogermavis »

creeksai wrote: post #91
rogermavis wrote: post #89

Earlier quoted context omitted.

> you are comparing it here to something that's always on That's the point. Our org was told databricks would solve problems we just didn't have. Serverful has some wonderful advantages: simplicity, (ironically) cheaper (than something running just 3-4 hours a day but which costs 10x), familiarity, reliability. Serverless also has advantages, but only if it runs smoothly, doesn't take an eternity to boot, isn't prohi…

I totally understand if Databricks doesn't fit your use cases. But you are doing a disingenuous comparison here because one can keep a "serverful" cluster up without shutting it down, and in that case, you'd never need to wait for anything to boot up. If you shut down your EC2 instances, it will also take time to boot up. Alternatively, you can use the (relatively new) serverless offering from them that gets you comp…

To ensure I'm not speaking incorrectly (as I was going from memory), I grep'ed my several years' of databricks notes. Oh boy.. the memories came flooding back!

We had 8 data engineers onboarding the org to databricks, it was only after 2 solid years before they got to working on serverless (it was because users complained of user unfriendliness of 'nodes', and managers of cost). But then, there were problems. A common pattern through my grep of slack convos is "I'm having this esoteric error where X doesn't work on serverless databricks, can you help".. a bunch of back and forth (sometimes over days) and screenshots followed by "oh, unfortunately, serverless doesn't support X".

Another interesting note is someone compared serverless databricks to bigquery, and bigquery was 3x faster without the databricks-specific cruft (all bigquery needs is an authenticated user and a sql query).

Databricks isn't useless. It's just a swiss army knife that doesn't do anything well, except sales, and may improve the workflows for the least advanced data analysts/scientists at the expense of everyone else.

Re: Databricks in talks to acquire startup Neon for about $1B

#94

Post by wqtz »

Databricks acquired bit.io and subsequently shut it down quite fast. Afaik bit.io had a very small team and the founder was a serial entrepreneur who is not going to stick around and he did not. I am not sure who from bit.io is still around at databricks.

If I am guessing right, Motherduck will likely be acquired by GCP because most of the founding team was ex-BQ. Snowflake purchased Modin and polars is still quite immature to be acquisition ready. So, what does this leave us with. There is also EDB who is competing in enterprise Postgres space.

Folks I know in the industry are not very happy with databricks. Databricks themselves was hinting people that that they would be potentially acquired by Azure as Azure tries to compete in the data warehouse space. But everyone become an AI company which left Databricks in an awkward space. Their bdev team is not bestest from my limited interactions with them (lots of starbucks drinkers and let me get back to you after a 3 month PTO), so they do not know who or how to lead them to an AI pivot. With cash to burn from overinvestment and the snowflake/databricks conf coming up fast they needed a big announcement and this is that big announcement.

Should have sobered up before writing this though. But who cares.

Re: Databricks in talks to acquire startup Neon for about $1B

#95

Post by fock »

mohon wrote: post #74
avg_dev wrote: post #71

Earlier quoted context omitted.

hmm, what is a serverless Pg? I don't quite understand. I thought you needed a database server if you wanted to run Pg.

basically they separate the compute and storage into different components, where the traditional PG use both compute and storage at the same server. because of this separation, the compute (e.q SQL parsing, etc) can be scaled independently and the storage can also do the same, which for example use AWS S3 so if your SQL query is CPU heavy, then Neon can just add more "compute" nodes while the "storage" cluster remain…

So how is this distributed Postgres still an ACID-compliant database? If you allow multiple nodes to query the same data this likely is just Trino/an OLAP-tool using Postgres syntax? Or did they rebuild Postgres and not upstream anything?

Re: Databricks in talks to acquire startup Neon for about $1B

#96

Post by walamaking »

rogermavis wrote: post #78
forgetfulness wrote: post #33

What is the lowdown on Databricks? Their bread and butter were hosted Spark and notebooks. As tasks done in Spark over a data lake began to be delegated wholesale to columnar store ELT, they tried to pivot to "lake houses", then I sort of lost track of them after I got out of Spark myself. Did Delta Lake ever catch on? Where are they going now?

It provides central place to store and query data. A big org might have a few hundred databases for various purposes - databricks lets data engineers set up pipelines to ETL that data into databricks and when the data is there it can be queried (using spark, so there's some downsides - namely a more restrictive SQL variant - but some advantages - better performance across very large datasets). Personally, I hated dat…

Dumb question - how is this different from Snowflake?

Re: Databricks in talks to acquire startup Neon for about $1B

#98

Post by fock »

th0ma5 wrote: post #88
woooooo wrote: post #54

Earlier quoted context omitted.

So, I'm not sure if this is less cynical or more cynical, but.. have you ever talked to the decision-makers who buy something like databricks? They can't build it themselves, and it's highly dubious that they'd be able to hire and supervise someone to build it. Databricks may be selling "nothing special", but it's needed, and the buyers can't build it themselves.

Is that how Databricks sees their customers? Yikes

I can tell you the company I work at (4000 people, legacy banking IT) has 4 people running our Datalake. We likely have more people buying/"evaluating" Databricks currently (from overhearing calls in open-plan offices), so I guess they have a point. A very sad point...

Re: Databricks in talks to acquire startup Neon for about $1B

#99

Post by levanten »

walamaking wrote: post #96
rogermavis wrote: post #78

Earlier quoted context omitted.

It provides central place to store and query data. A big org might have a few hundred databases for various purposes - databricks lets data engineers set up pipelines to ETL that data into databricks and when the data is there it can be queried (using spark, so there's some downsides - namely a more restrictive SQL variant - but some advantages - better performance across very large datasets). Personally, I hated dat…

Dumb question - how is this different from Snowflake?

They are very similar; with various similar solutions at differing stages of maturity.

Re: Databricks in talks to acquire startup Neon for about $1B

#100

Post by gopalv »

newfocogi wrote: post #4

They offer serverless Postgres. Here's a link if anyone else needs it https://neon.tech/

An OLTP solution fixes a lot of the headaches about the traditional extract-load-transform steps.

Mostly a lot of OLAP starts when the data loads in Kafka logs or a disk of some sort.

Then you schedule a task or keep a task polling this constantly, which is always prone to small failures & delays or big failures when schema changes up.

The "data pipeline" team exists because the data doesn't move by itself from where it is first stored to where it is ready for deep analysis.

If you can directly push 1-row updates transactionally to a system and feed off the backend to write a more OLAP friendly structure, then you can hookup things like a car rental service's operational logs into a system which can compute more complex things like forecasting of availability or apply discounts to give a customer an upgrade for cheap.

Neon looks a lot better than YugaByte in tech (which also talks postgres protocols) and a lot nicer in protocol compatibility than something like FoundationDB.

Alloy from Google feels somewhat similar, Spanner has a postgres interface too.

The postgres API is a great abstraction common point, even if the actual details of the implementations vary a lot.

Post reply on HN