Viewing profile — dmpetrov

[flagged]

[flagged]

I like the idea a lot but it's still unclear from the docs what the hard security boundary is once you start calling LLMs - can it avoid "breaking out" into the host env in practic…

Good point - it does sound a bit like marketing bs. We’ll rephrase it.

Please share your feedback!

Good question! I’m not so familiar with it. It looks like Daft is closer to Lance with it’s own data format and engine. But I’d appreciate more insights from users or the creators.…

Lance is just a data format. Lance DB might be more comparable to DataChain. DataChain focuses on data transformation and versioning, whereas LanceDB appears to be more about retri…

It's not a format :) It's simpliy about linking metadata from a json to a corresponding image or video file, like pairing data003.png & data003.json to a single, virtual record. So…

Absolutely, that's a common scenario! Just connect from your Python code (like the lambda in the example) to DB and extract the necessary data.

I guess, it involves splitting a file into smaller document snippets, getting page numbers and such, and calculating embeddings for each snippet—that’s the usual approach. Specific…

Exactly! DataChain does lazy compute. It will read metadata/json while applying filtering and only download a sample of data files (jpg) based on the filter. This way, you might en…

DataChain has no assumptions about metadata format. However, some formats are supported out of the box: WebDataset, json-pair, openimage, etc. Extract metadata as usual, then retur…

Yes, it's not meant to replace data engineering tools like Prefect or Temporal. Instead, it serves as a transformation engine and ad-hoc analytics for images/video/text data. It's …

Yay! Excited to see DataChain on the front page :) Maintainer and author here. Happy to answer any questions. We built DataChain because our DVC couldn't fully handle data transfor…

Can this work statistically? For a giving number of attempts, you can ger a required number of successes to make sure it's a statistically meaningful result. In theory, this approa…

Thank you! Just shoot an email to support and mention HN. I’ll read and reply.

Right, DVC caches data for consistency and reproducibility. If caching is not needed and streaming required, we've created a sister tool DataChain. It's even supports WebDataset an…

Yes. And if you track transformations of the binaries or ml training