why not a multimodal embedding model?
How we index images for RAG
11–20 of 41 posts
Re: How we index images for RAG
#12Well I don't know if this one has been getting by others too but I have been doing this since 2 years ago and it works really well. Except the fact that for the documents I had to chunk containing these images I had to chase the authors(multiple of them) to update the relevant captions for their images.
It is cost efficient than multi-modal. Lesser ingestion time altogether. Only part is that if the retrieval query is a question which can be answered only after looking at the image, then this architecture would need some little modification.
Re: How we index images for RAG
#13why not a multimodal embedding model?
The article do mentions why they don't use multimodal retrieval. Also I think this approach is cheaper (compute wise) than multimodal retrieval. From the article:
Multimodal retrieval does not suit this domain. CLIP-style embeddings wash out exactly the fine detail that matters in charts, tables, and annotated screenshots, and short technical queries ("how do I configure X") give too little signal to match against image vectorsRe: How we index images for RAG
#14why not a multimodal embedding model?
Article says this misses important details, eg data that might be in the image.
Re: How we index images for RAG
#15Re: How we index images for RAG
#16How descripting is the caption that you obtain?
So you include colour, shapes, etc?
Re: How we index images for RAG
#17> we don't send images to the model at query time. We describe each image once, at indexing time, with a cheap vision model, store the descriptions as text, and retrieve them alongside ordinary text chunks This is what I've been doing in my Obsidian infodump for a while. If I know that an image is important, I generate a text description (Mermaid if possible, English if not) and paste it after the image in a block. T…
What does Mermaid text description of an image mean?
Descriptions of images that are charts or diagrams to start with?
Re: How we index images for RAG
#18[flagged]
Re: How we index images for RAG
#19[flagged]
Re: How we index images for RAG
#20[flagged]