Viewing profile — sammysidhu

Amazing work leveraging Daft for this!

Part of the Daft team here! Happy to answer any questions

Hi, One of the authors of Daft here! Happy to answer any questions

Congrats on the launch! It's been great working with you from the Daft side

https://github.com/Eventual-Inc/Daft Is also great at these types of workloads since it’s both distributed and vectorized!

We're actually using pyiceberg to retrieve metadata! All our IO and decoding happens in the rust side once the data has been passthrough. We expose something called a ScanOperator …

Eventual | Engineers; Rust | SF | https://www.ycombinator.com/companies/eventual/jobs We're the people behind Daft (On the front page today!), a distributed dataframe built in Rust…

we love rust too!

This would be more comparable to being a backend for ibis. We're working on adding the remaining operations (like regex on strings or trigonometry functions) that ibis requires!

horizontal scaling however provides you with more aggregate network bandwidth. Most enterprises run workloads that downloads data from a data lake (usually S3), which is usually th…

One of the daft developers here - Thank you!

Love the data driven approach to massive problem

(one of the Daft maintainers here) Great call out! I went ahead and make an issue for us to work on this: https://github.com/Eventual-Inc/Daft/issues/1016

Hi! (one of the Daft maintainers here), thanks for the feedback. Ultimately you're right that supporting the full Polars syntax in a distributed fashion is very difficult. There ar…

Hi (one of the maintainers here), that is a good suggestion! I wasn't aware of that project. I went ahead and made an issue to add `export DO_NOT_TRACK=1` as one of the variables w…

Hi Sammy here, one of the Daft maintainers. Happy to answer any questions that you all might have!