191–200 of 234 posts
Interesting trend - modern serverless databases choosing Rust for its memory safety, performance predictability. Makes sense for systems where reliability and efficiency are non-negotiable.
I believe that future data platforms will adopt an all-in-one approach, offering OLTP, OLAP, as well as support for other hybrid workloads such as vector, graph, and time series. This will lower user costs and be more friendly to applications in the AI era.
Earlier quoted context omitted.
Not commoditising for enterprise. My last gig wouldn’t allow open source software or any company that might not be there in a decade, or which kept data anywhere but our own tenant. We’d look for the “call us” pricing rather than hate it, which I normally do. We added databricks and it was considered one of my top three achievements, because they don’t have to think about data platforms again, just focus on using it.…
Enterpise also often wants a full data platform (like Databricks), not a plain data warehouse.
I applied to neon last week and then the news broke about the acquisition. They rejected it this morning — I have never been happier to receive a rejection to an application. This would’ve been three acquisitions straight for me and… I’m okay, they’re awful. I just want stability. Congrats to the neon team! I use and love neon. Really hope this doesn’t change them too much.
I got hired at Kenna Security a month before they were acquired by Cisco and it was such a miserable experience that I won't work for any company the Kenna leadership are involved with, nor would I ever consider working at Cisco.
Earlier quoted context omitted.
Not commoditising for enterprise. My last gig wouldn’t allow open source software or any company that might not be there in a decade, or which kept data anywhere but our own tenant. We’d look for the “call us” pricing rather than hate it, which I normally do. We added databricks and it was considered one of my top three achievements, because they don’t have to think about data platforms again, just focus on using it.…
Enterpise also often wants a full data platform (like Databricks), not a plain data warehouse.
Earlier quoted context omitted.
Aren't the alternatives you mentioned - icerberg and duckdb - both storage solutions while spark is a way to express distributed compute? I'm a bit out of touch with this space, is there a newer way to express distributed compute?
duckdb is primarily a query engine. It does have a storage format, but one of it's strengths is querying data where it already resides (e.g. a parquet file sitting in S3). There are some examples[0] of enabling DuckDB to manage distributed workloads, but these are pretty experimental. 0 - https://www.definite.app/blog/smallpond
Earlier quoted context omitted.
Enterpise also often wants a full data platform (like Databricks), not a plain data warehouse.
I'm curious - as someone who hasn't been paying super close attention to the space, what does it mean to be a full data platform? Is it just having all the different flavors of DB you might need all from one vendor, or is there also tighter integration than if you cobbled it together from multiple vendors?
It’s basically a luxury minivan. It’s may not be the fastest or prettiest or cheapest, but it’s a safe way for a large family of “data and AI people” to traverse a large organisation.
More seriously, I like to call it an “analytics workbench” in a professional setting.
Maybe unrelated but Databricks is the most annoying garbage I have ever had to use. It fascinates me how anyone uses it by choice.
Databricks' proprietary notebook format that introduced subtle incompatibilities with Jupyter was infuriating embrace-extend-extinguish style bullshit, but on-prem cluster instability causing jobs to crash on a daily basis was way more infuriating, and at that time, enterprises were more than happy to pay a premium to accelerate analytics teams.
In the 2010s, Databricks had a solid billion-dollar business. But Spark-as-a-Service by itself was never going to be a unicorn idea. AWS EMR was the giant tortoise lurking in the background, slowly but surely closing the gap. The status quo couldn't hold, and who doesn't want to be a unicorn? So, they bloated the hell out of the product, drank that off-brand growth-hacker Kool-Aid, and started spewing some of the most incoherent buzz-word salad to ever come out of the Left Coast. Just slapping data, lake, and house onto the ends of everything, like it was baby oil at a Diddy Party.
Now, here we are in 2025, deep into the terminal decline of enshittification, and they're just rotting away, waiting for One Real Asshole Called Larry Ellison to scoop them up and take them straight to Hell. The State of Florida, but for Big Data companies.
It would be a mystery to me too, why anyone would pick Databricks today for a greenfield project, but those enterprises from 5+ years ago are locked in hard now. They'll squeeze those whales and they'll shit money like a golden goose for a few more years, but their market share will steadily decrease over the next few years.
It's the cycle of life. Entropy always wins. Eventually the Grim Reaper Larry comes for us all. I wouldn't hate on them too hard. They had a pretty solid run.
Data warehousing is quickly becoming a commodity through open-source. I know a company who had 2PBs+ of data in Cloudera. But instead of moving to the cloud (and Databricks), they saved 5X costs by building their own analytics platform with Iceberg, Trino and Superset. The k8s operators are enterprise quality now. On-premises S3 is good, too. You can have great hardware (servers with 128 cpus and 1 TB) and networking…
Earlier quoted context omitted.
Totally agree. Happy open source StarRocks user here using the k8s operator for customer-facing analytics on terabytes of data. There's very little need for Databricks in my world.
Looking at StarRocks site ( https://www.starrocks.io/ ), they compare against Clickhouse, Druid and Trino. Don't even compare against Spark/Databricks! Guess Spark is just not competitive.