11–20 of 27 posts
Cool! Does this assume the unstructured data already has a corresponding metadata file? My most common use cases involve getting PDFs or HTML files and I have to parse the metadata to store along with the embedding. Would I have to run a process to extract file metadata into JSONs for every embedding/chunk? Would keys created based off document be title+chunk_no? Very interested in this because documents from clients…
Cool! Does this assume the unstructured data already has a corresponding metadata file? My most common use cases involve getting PDFs or HTML files and I have to parse the metadata to store along with the embedding. Would I have to run a process to extract file metadata into JSONs for every embedding/chunk? Would keys created based off document be title+chunk_no? Very interested in this because documents from clients…
DataChain has no assumptions about metadata format. However, some formats are supported out of the box: WebDataset, json-pair, openimage, etc. Extract metadata as usual, then return the result as JSON or a Pydantic object. DataChian will automatically serialize it to internal dataset structure (SQLite), which can be exported to CSV/Parquet. In case of PDF/HTML, you will likely produce multiple documents per file whic…
Could your metadata come from something like a Postgres sql statement? Or an iceberg view?
Cool! Does this assume the unstructured data already has a corresponding metadata file? My most common use cases involve getting PDFs or HTML files and I have to parse the metadata to store along with the embedding. Would I have to run a process to extract file metadata into JSONs for every embedding/chunk? Would keys created based off document be title+chunk_no? Very interested in this because documents from clients…
What relevant metadata is there in an HTML file?
Hopefully, @jerednel can add more details.
Earlier quoted context omitted.
DataChain has no assumptions about metadata format. However, some formats are supported out of the box: WebDataset, json-pair, openimage, etc. Extract metadata as usual, then return the result as JSON or a Pydantic object. DataChian will automatically serialize it to internal dataset structure (SQLite), which can be exported to CSV/Parquet. In case of PDF/HTML, you will likely produce multiple documents per file whic…
> DataChain has no assumptions about metadata format. Could your metadata come from something like a Postgres sql statement? Or an iceberg view?
Just connect from your Python code (like the lambda in the example) to DB and extract the necessary data.
Cool! Does this assume the unstructured data already has a corresponding metadata file? My most common use cases involve getting PDFs or HTML files and I have to parse the metadata to store along with the embedding. Would I have to run a process to extract file metadata into JSONs for every embedding/chunk? Would keys created based off document be title+chunk_no? Very interested in this because documents from clients…
DataChain has no assumptions about metadata format. However, some formats are supported out of the box: WebDataset, json-pair, openimage, etc. Extract metadata as usual, then return the result as JSON or a Pydantic object. DataChian will automatically serialize it to internal dataset structure (SQLite), which can be exported to CSV/Parquet. In case of PDF/HTML, you will likely produce multiple documents per file whic…
Forgive my ignorance, but what is "json-pair"?
Earlier quoted context omitted.
DataChain has no assumptions about metadata format. However, some formats are supported out of the box: WebDataset, json-pair, openimage, etc. Extract metadata as usual, then return the result as JSON or a Pydantic object. DataChian will automatically serialize it to internal dataset structure (SQLite), which can be exported to CSV/Parquet. In case of PDF/HTML, you will likely produce multiple documents per file whic…
> However, some formats are supported out of the box: WebDataset, json-pair, openimage, etc. Forgive my ignorance, but what is "json-pair"?
It's simpliy about linking metadata from a json to a corresponding image or video file, like pairing data003.png & data003.json to a single, virtual record. Some format use this approach: open-image or laion datasets.
Earlier quoted context omitted.
What relevant metadata is there in an HTML file?
I guess, it involves splitting a file into smaller document snippets, getting page numbers and such, and calculating embeddings for each snippet—that’s the usual approach. Specific signals vary by use case. Hopefully, @jerednel can add more details.
My retriever functions will typically use metadata in combination with the similarity search to do impart some sort of influence or for reranking.
How does this relate to https://github.com/lancedb/lance
DataChain focuses on data transformation and versioning, whereas LanceDB appears to be more about retrieving and serving data. Both designed for multimodal use cases.
From technical side: Lance has it's own data format and DB engine while DataChain utilizes existing DB engines (SQLite in open-source and ClickHouse/BigQuery in SaaS).
In SaaS, DataChain has analytics features including data lineage tracking and visualization for PDFs, videos, and annotated images (e.g., bounding boxes, poses). I'm curious to understand the unique value of LanceDB's SaaS — insight would be helpful!
You could think of it as OLTP (Lance) versus OLAP (DataChain) for multimodal data, though this analogy may not be perfect.
How does this relate to https://github.com/lancedb/lance
Lance is just a data format. Lance DB might be more comparable to DataChain. DataChain focuses on data transformation and versioning, whereas LanceDB appears to be more about retrieving and serving data. Both designed for multimodal use cases. From technical side: Lance has it's own data format and DB engine while DataChain utilizes existing DB engines (SQLite in open-source and ClickHouse/BigQuery in SaaS). In SaaS,…