Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we index images for RAG

kapa.ai

Post reply on HN
Print view View original

21–30 of 41 posts

Re: How we index images for RAG

#21

Post by bad_username »

Terretta wrote: post #17
bad_username wrote: post #4

> we don't send images to the model at query time. We describe each image once, at indexing time, with a cheap vision model, store the descriptions as text, and retrieve them alongside ordinary text chunks This is what I've been doing in my Obsidian infodump for a while. If I know that an image is important, I generate a text description (Mermaid if possible, English if not) and paste it after the image in a block. T…

What does Mermaid text description of an image mean? Descriptions of images that are charts or diagrams to start with?

Most diagrams I come across are basically boxes and arrows which are representable with mermaid flow charts without losing information. The layout of the mermaid will usually look differently, but that is not typically what matters. ChatGPT is quite good in creating mermaid flow charts from random box and arrow diagram images.

Re: How we index images for RAG

#23

Post by vinzenzu »

bad_username wrote: post #4

> we don't send images to the model at query time. We describe each image once, at indexing time, with a cheap vision model, store the descriptions as text, and retrieve them alongside ordinary text chunks This is what I've been doing in my Obsidian infodump for a while. If I know that an image is important, I generate a text description (Mermaid if possible, English if not) and paste it after the image in a block. T…

For a RAG project for a client with a lot of PDFs and Powerpoints with images, I used ColPali a year ago. I see the provider ColiVara is still online but it seems to have fizzled out.

Retrieving based on text and then giving the generation model the image instead is much smarter than retrieving based on image. Image-based retrieval is slow and expensive.

Same with giving the model an image vs a structured representation of it.

Re: How we index images for RAG

#24

Post by emil_sorensen »

breadislove wrote: post #15
efavdb wrote: post #14

Earlier quoted context omitted.

Article says this misses important details, eg data that might be in the image.

very bad take. with most modern multomodal models you get way better performance then going to text first

it's a cost/latency trade-off in production + very use-case dependent

Re: How we index images for RAG

#26

Post by throwaw12 »

vinzenzu wrote: post #23
bad_username wrote: post #4

> we don't send images to the model at query time. We describe each image once, at indexing time, with a cheap vision model, store the descriptions as text, and retrieve them alongside ordinary text chunks This is what I've been doing in my Obsidian infodump for a while. If I know that an image is important, I generate a text description (Mermaid if possible, English if not) and paste it after the image in a block. T…

For a RAG project for a client with a lot of PDFs and Powerpoints with images, I used ColPali a year ago. I see the provider ColiVara is still online but it seems to have fizzled out. Retrieving based on text and then giving the generation model the image instead is much smarter than retrieving based on image. Image-based retrieval is slow and expensive. Same with giving the model an image vs a structured representat…

> For a RAG project for a client with a lot of PDFs and Powerpoints with images, I used ColPali a year ago

How was the accuracy compared to pre-parsing the image and doing search in the text?

Re: How we index images for RAG

#29

Post by kaibuidsai »

hparadiz wrote: post #7

With media ingestion this is called "eager" processing. Historically for things like pulling thumbnails for images / video and pre-generating common sizes for things. This follows the same pattern and makes all the sense in the world. My only concern is that due to the non deterministic nature of LLMs new models will reveal new information about your data. For example you might identify a car in an image but the cont…

preprocess the image with vlm was an old common practice
Post reply on HN