Viewing profile — jaychia

Very cool that this kind of work can now be performed at this kind of a price-point. 24 hours for 8M filings on just 12 cores :) Excited for unstructured/multimodal data processing…

It's fairly surprising to me how naive/early we are still in the techniques that we use here. Anthropic's post on the Claude Agent SDK (formerly Claude Code SDK) talks about how th…

Great article and a timely reminder for many :) Applicable not just for grad school applications, but also to job apps, startups, and relationships. Hang in there y'all, all it tak…

Check out Daft (www.getdaft.io) - we've been working really hard on our Iceberg support. Supports full reads/writes (including partitioned writes) and our SQL support is also comin…

Hey, I'm one of the developers of Daft :) Thanks for the feedback on marketing! Daft is indeed distributed using Ray, but to do so involves Daft being architected very carefully fo…

One of the maintainers of Daft here. Just dug through the datachain codebase to understand a little more. I think while both projects have a Dataframe interface , they're very diff…

I work on Daft and we’ve been collaborating with the team at Amazon to make this happen for about a year now! We love Ray, and are excited about the awesome ecosystem of useful + s…

There’s a lot of interesting work happening in this area (see: XTable). We are building a Python distributed query engine, and share a lot of the same frustrations… in fact until q…

Interesting. Daft currently does validation on types/names only at runtime. The flow looks like: 1. Construct a dataframe (performs schema inference) 2. Access (now well-typed) col…

Daft developer here! We actually already have read support. Check out the pyiceberg docs' Daft section: https://py.iceberg.apache.org/api/#daft It's also very easy to use from Daft…

Oh yes good point! We'll be sure to add more details about comparisons with local dataframe libraries such as Pandas/Polars/DuckDB.

Hello, Daft developer here! The network indeed becomes the bottleneck. In 2 main ways: 1. Reading data from cloud storage is very expensive. Here’s a blogpost where we talk about s…

Hello! Daft developer here. We are most similar in API to Polars and PySpark. And thanks for the feedback! We’ll add more capabilities for regex, as well as flesh out our documenta…

Hello! Daft developer here. The benchmarks we performed aren’t directly comparable to the benchmarks on TPC-H’s own page because of differences in hardware, storage etc. For hardwa…

Hello! Daft developer here - we don’t directly use Polars as an execution engine, but parts of the codebase (e.g. the expressions API) are heavily influenced by Polars code and hen…

Spent some time diving into the Apache Parquet file format, which was surprisingly complicated and nuanced. There's lots of lore/history in the versioning of the format's various f…

> Ray backend runner Yes, give it a whirl and let us know what you think! Ray is amazing and has actually gotten a lot better post their 2.0 release :) > Is this based on Apache Ar…

We hear you, and thanks for making this visible! As a performance-driven project it’s important for us to understand which operations and use-cases are slowest/buggiest for our use…

> Does Daft support Delta table format? Not yet, it’s on our todo list to integrate with the ecosystem of data catalogs (Iceberg/Delta/Hudi etc). Join our Slack/get in touch with u…

> So Daft is a distributed Polars ? We did actually start by using Polars as our underlying execution engine, but eventually transitioned off to our own Rust Table abstraction to b…

Hi, I'm one of the maintainers of Daft 1. Thanks! We think so too :) 2. Here's my 2c in argument of flat files - Ingestion: ingesting things into a data lake is much easier than wr…

Hello! I am one of the maintainers of Daft. Funny enough I just gave a presentation about Daft in London and we all had quite a laugh at the name :D