71–80 of 112 posts
Earlier quoted context omitted.
650GB is something one could handle using a local filesystem, no need for complex tooling.
650 GB fits in ram: https://yourdatafitsinram.net
Even Trino runs circles around Spark, with some heavier jobs simply not completing in Spark at all (total data size up to a single PB, with about 10TB of RAM available for compute), and Trino isn't known for its extreme performance. StarRocks is noticeably faster still, so I wouldn't right off distributed compute just yet- at least for some applications.
And even then, performance isn't the most important criterion for an analytics tool choice- more probably depends on the integrations, access control, security, ease of extendability, maintenance, scaling, support by existing instruments. Boring enterprise stuff, sure, but for those older frameworks it's all either readily available, or can be quickly added with little experience (writing a java plugin for Trino is as easy as it gets).
With Duckdb or Polars (if used as a basis for a datalake/house etc) it may degrade into an entire team of engineers wasting resources on implementing the tooling around the tooling instead of providing something actually useful for the business
650GB? Your data is small, fits on my phone. Dump the hyped tooling and just use gnu tools. Here's an oldie on the topic: https://adamdrake.com/command-line-tools-can-be-235x-faster-...
What is the point of simulating 650GB data with ~40 columns if you are going to use a single column for testing? Is that even 16GB?
DuckDb has a new "DuckLake" catalog format that would be another candidate to test. https://ducklake.select/
What is the point of simulating 650GB data with ~40 columns if you are going to use a single column for testing? Is that even 16GB?
It's a strided array and slows down memory access.
In places I have worked at that used Databricks, I feel they chose it for the same reasons big orgs use Microsoft: it comes out of a box and has a big company behind it. Technical benchmarks or even cost considerations would be a distant second.
There are real advantages from having a managed data platform compared to managing everything yourself, especially if you have a large number of data teams that need to collaborate.
Presto (a.k.a. AWS Athena) might be a faster/better alternative? Also would like to see if 650GB data is available locally.
Presto is renamed to Trino now. But I concur with what you say. It is also very cheap in both maintenance and running cost. It is just an amazing tool and you pay (RIP) pennies.
650GB? Your data is small, fits on my phone. Dump the hyped tooling and just use gnu tools. Here's an oldie on the topic: https://adamdrake.com/command-line-tools-can-be-235x-faster-...
What if it was 650TB? This article is obviously a microbenchmark. I work with much larger datasets, and neither awk nor DBD would make a difference to the overall architecture. You need a data catalog, and you need a clusters of jobs at scale, regardless of a data format library, or libraries.