Earlier quoted context omitted.
I think that's the main offering of databricks- you get a "data platforn in a box" and navigating the forest of piecemeal solutions is replaced with telling your data science and analytics teams to "use databricks". It's easy to look on knowing lots about data tools and say "this could be better done with open source tools for a fraction of the cost", but if you're not a big tech company, hiring a team to manage your…
What exactly is a "data platform"? We have a large postgres server running on a dedicated server that handles millions of users, billions of record updates and inserts per day, and when I want to run an analysis I just open up psql. I wrote some dashboards and alerting in python that took a few hours to spin up. If we ever ran into load issues, we'd just set up some basic replication. It's all very simple and can eas…
Imagine you're a big company with loads of teams/departments multiple different types of SQL servers for data reporting, plus some parquet datalakes, and hey, just for fun why not a bunch of csvs.
Getting data from all these locations becomes a full time job, so at some point someone wants some tool/ui that lets data analysts log into a single thing, and get the experience that you currently have with one postgres server.
I think it's not a problem of scale in the CS sense, more the business sense where big organisations become complex and disorganised and need abstractions on top to make them workable.