650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

Post by jellyfishbeaver » Sun, Nov 16, 2025, 2:57 PM UTC

esafak wrote: post #9

mynameisash wrote: post #8

Earlier quoted context omitted.

> It seems like these single-node libraries can process a terabyte on a typical machine, and you'd have have over 10TB before moving to Spark. I'm surprised by how often people jump to Spark because "it's (highly) parallelizable!" and "you can throw more nodes at it easy-peasy!" And yet, there are so many cases where you can just do things with better tools. Like the time a junior engineer asked for help processing 1…

I used pySpark some time ago when it was introduced to my company at the time and I realized that it was slow when you used python libraries in the UDFs rather than pySpark's own functions.

We actually baked a rule to catch UDF usage into our Python linter. Almost always, a UDF can be refactored to use only native PySpark functions.

Post by jellyfishbeaver » Sun, Nov 16, 2025, 3:04 PM UTC

pu_pe wrote: post #45

In places I have worked at that used Databricks, I feel they chose it for the same reasons big orgs use Microsoft: it comes out of a box and has a big company behind it. Technical benchmarks or even cost considerations would be a distant second.

Are there really any other managed Spark/full data platform providers that come close to the level of ergonomics and maturity of Databricks? I agree some of their features are half-baked and you can turn on some dangerous cash faucets if you don't have a very diligent administrator, but I have not come across anything that our developers (fresh grads up to seasoned Spark veterans) liked as much as Databricks.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark