21–27 of 27 posts
Earlier quoted context omitted.
> However, some formats are supported out of the box: WebDataset, json-pair, openimage, etc. Forgive my ignorance, but what is "json-pair"?
It's not a format :) It's simpliy about linking metadata from a json to a corresponding image or video file, like pairing data003.png & data003.json to a single, virtual record. Some format use this approach: open-image or laion datasets.
Earlier quoted context omitted.
Lance is just a data format. Lance DB might be more comparable to DataChain. DataChain focuses on data transformation and versioning, whereas LanceDB appears to be more about retrieving and serving data. Both designed for multimodal use cases. From technical side: Lance has it's own data format and DB engine while DataChain utilizes existing DB engines (SQLite in open-source and ClickHouse/BigQuery in SaaS). In SaaS,…
How about daft https://github.com/Eventual-Inc/Daft - also looks like a new multimodal dataframe framework
It looks like Daft is closer to Lance with it’s own data format and engine. But I’d appreciate more insights from users or the creators.
Wow! Great instrument, so excited to see it here
I’m not sure if this term postmodern data stack was invented for the purposes of this copy. Probably not. But terms like this don’t really engender a lot of faith that this isn’t yet another piece of the now decades long hype cycle data engineering products face
Earlier quoted context omitted.
Lance is just a data format. Lance DB might be more comparable to DataChain. DataChain focuses on data transformation and versioning, whereas LanceDB appears to be more about retrieving and serving data. Both designed for multimodal use cases. From technical side: Lance has it's own data format and DB engine while DataChain utilizes existing DB engines (SQLite in open-source and ClickHouse/BigQuery in SaaS). In SaaS,…
How about daft https://github.com/Eventual-Inc/Daft - also looks like a new multimodal dataframe framework
Just dug through the datachain codebase to understand a little more. I think while both projects have a Dataframe interface, they're very different projects!
Datachain seems to operate more on the orchestration layer, running Python libraries such as PIL and requests (for making API calls) and relying on an external database engine (SQLite or BigQuery/Clickhouse) for the actual compute.
Daft is an actual data engine. Essentially, it's "multimodal BigQuery/Clickhouse". We've built out a lot of our own data system functionality such as custom Rust-defined multimodal data structures, kernels to work on multimodal types, a query optimizer, distributed joins etc.
In non-technical terms, I think this means that Datachain really is more of a "DBT" which orchestrates compute over an existing engine, whereas Daft is the actual compute/data engine that runs the workload. A project such as Datachain could actually run on top of Daft, which can handle the compute and I/O operations necessary to execute the requested workload.
> Datachain does not abstract or hide the AI models and API calls, but helps to integrate them into the postmodern data stack. I’m not sure if this term postmodern data stack was invented for the purposes of this copy. Probably not. But terms like this don’t really engender a lot of faith that this isn’t yet another piece of the now decades long hype cycle data engineering products face