Not super related but one complaint about Polars is I don't get why the departure from the pandas API...
650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark
101–110 of 112 posts
Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark
#102I love this article! But I think this insight shouldn't be surprising. Distribution always has overheads, so if you can do things on a single machine it will almost always be faster. I think a lot of engineers expect 100 computers to be faster than 1, because of the size comparison. But we're really looking at a process here, and a process shifting data between machines will almost always have to do more stuff, and t…
The Scalability at what COST paper (pdf https://www.usenix.org/system/files/conference/hotos15/hotos...) is my favorite thing. Single worker implementation wipes the floor with big distributed solutions.
Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark
#103would like to see work like this, but for datasets in the hundreds of TB or single-digit PB but i definitely agree about this point > Cluster fatigue is real imo, the concept of “extremely ephemeral query workers” is under-explored stateless, maintenance-free, burstable fleets of query workers is what I would like to see more of in the future. it’s how we do it, and it gives us full-text search on multi-hundred terab…
Yes.. its called snowflake? Theyre exactly that and why they work so well. I know youre asking for an OSS but what snowflake offers is a fleet of servers that can build your cluster in a second as opposed to minutes that you need if you want to spin it up yourself..
Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark
#104It's cool that this is possible on a single node but I still think distributed is the way.
The point of these tools is productivity. What are you trying to accomplish and how long does it take to accomplish? This includes time spent writing code and fussing with configs. This would take Yes cost matters also, but running many machines for a short period of time is the same as one for a long time? Open to honest rebuttal.
Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark
#105I had to do something like this for a few TB of json recently. The unique thing about this workload was it was a ton of small 10-20mb files. I found that clickhouse was the fastest, but duckdb was the simplest to work with it usually just works. DuckDB was close enough to the max performance from clickhouse. I tried flink & pyspark but they were way slower (like 3-5x) than clickhouse and the code was kind of annoying…
Did you first ingest/convert this data to some other format, or did you operate directly on the JSON?
Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark
#106Earlier quoted context omitted.
c5 is such a bad instance type, m6a would be so much better and even cheaper, I would love to see this on an m8a.2xlarge (7th and 8th generations don’t use SMT) and that is even cheaper and has up to 15 Gbps
Actually for this kind of workload 15Gbps is still mediocre. What you actually want is the `n` variant of the instance types, which have higher NIC capacity. In the c6n and m6n and maybe the upper-end 5th gens you can get 100Gbps NICs, and if you look at the 8th gen instances like the c8gn family, you can even get instances with 600Gbps of bandwidth.
The math here is weird.
A Samsung 990 Pro reads at something like 50 Gbps and PCIe 4.0 x4 is quite a bit faster than that. You can get this speed with a queue depth that isn’t crazy, and you can have multiple NVMe operations in flight reading the same large Parquet file. Latency is in the tens of microseconds.
The consensus seems to be that S3 can read one object at somewhat under 1Gbps. You can probable scale that to the full speed of your NIC by reading multiple objects at once, but you may not be able to scale by reading one object in multiple overlapping ranges. Latency is in the milliseconds.
So, sure, an EC2 with a fast instance and massive multiple object parallelism can have 10x higher bandwidth than an NVMe device, but the amount of parallelism and latency tolerance needed is a couple orders of magnitude higher than NVMe. Meanwhile that NVMe device does not charge for read operations and costs a couple hundred dollars, once.
If you are so inclined, you can build an NVMEoF setup (at much much higher cost) that separates compute and storage and has excellent performance, but this is a nontrivial undertaking.
Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark
#107Not super related but one complaint about Polars is I don't get why the departure from the pandas API...
I was very reluctant about the polars syntax as well initally, but it has grown a lot on me.
Pandas syntax is super ergonomic for quick one-off analysis, but it becomes hard to read/maintain once your processing gets more complex.
For example, the innocent
df[arg]
In contrast, polars syntax feals clunky initially, but it's much easier for me to revisit a pipeline and quickly understand what it does.
Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark
#108Hey everyone, I'm a software engineer at Eventual, the team behind Daft! Huge thanks to the op for the benchmark, we're a huge fan of your blog posts and this gave us some really useful insights. For context, Daft is a high-performance data processing engine for AI workloads that works both on single-node and distributed setups. We're actively looking into the results of the benchmark and hope to share some of our fi…
Do you plan to expose daft as a backend in ibis?
That would be the best way to smoothly test it out and transition workloads from other engines for codebases in my teams.
Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark
#109would like to see work like this, but for datasets in the hundreds of TB or single-digit PB but i definitely agree about this point > Cluster fatigue is real imo, the concept of “extremely ephemeral query workers” is under-explored stateless, maintenance-free, burstable fleets of query workers is what I would like to see more of in the future. it’s how we do it, and it gives us full-text search on multi-hundred terab…
> extremely ephemeral query workers
Reading data from S3 can really add up, so this isn't as straightforward as it seems.
Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark
#110This is most misrepresented article on two fronts 1. tested column pruning and the dataset you access would have been 2 columns + metadata for the parquet files so probably fit in memory even without streaming. 2. Most of the processing time would be IO bound on S3 and the access patterns/simultaneous connection limits etc. would have more of an impact than any processing code. Love that you went through the pain of…
1. Important points that the query is a projection that only returns a fraction of the 650GB that fits in memory. DuckDB is good at streaming larger than memory queries, Polars less mature there. That would show in the results.
2. S3 defaults shouldn't prevent all available threads/cpus from reading the files in parallel, so I would assume that the network bandwidth of the VM (or container) would be the bottleneck.