Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs, RAG, and the missing storage layer for AI

blog.lancedb.com

Post reply on HN
Print view View original

41–50 of 64 posts

Re: LLMs, RAG, and the missing storage layer for AI

#41

Post by sgt101 »

brigadier132 wrote: post #9
panarky wrote: post #4

The first unstated assumption is that similar vectors are relevant documents, and for many use cases that's just not true. Cosine similarity != relevance. So if your pipeline pulls 2 or 4 or 12 document chunks into the LLM's context, and half or more of them aren't relevant, does this make the LLM's response more or less relevant? The second unstated assumption is that the vector index can accurately identify the top…

The vectors are literally constructed so that cosine similarity is semantic similarity. > second unstated assumption is that the vector index can accurately identify the top K vectors by cosine similarity, and that's not true either Its not unstated, its called ANN for a reason

yes - but calculating the consine similarity for all the candidates is prohibitively expensive.

hence heuristic.

Re: LLMs, RAG, and the missing storage layer for AI

#42

Post by reissbaker »

BoorishBears wrote: post #39
reissbaker wrote: post #32

Earlier quoted context omitted.

I've had decent results using a doc2query style approach: 1. Ask an LLM to return a list of questions answered by the document 2. Store the embeddings of the questions along with a document ID 3. On user query, get the embedding of the user query 4. KNN cosine similarity search the user embedding vs. the corpus of question embeddings 5. Return the highest ranked documents You can tweak this approach depending on your…

You just reinvented a 2 year old technique with a more expensive pipeline and missed performance gains (from the cross-encoder step): https://www.sbert.net/examples/domain_adaptation/README.html https://arxiv.org/abs/2112.07577

I'm aware of more efficient ways to do it! (Hence referencing e.g. doc2query.) But you have to train a model, whereas with an LLM you can get a working version in 5mins of work.

Re: LLMs, RAG, and the missing storage layer for AI

#43

Post by spott »

regularfry wrote: post #34
spott wrote: post #17

Earlier quoted context omitted.

You probably aren’t using an LLM for your text embeddings for document retrieval (they don’t perform as well as specialist embedding models[0]), and even if they did, you have an embedding about a bare document, without any context of what you are trying to get out of it. If you were to add your context in and then get an embedding, you would get a different answer. As your query gets specific, irrelevant aspects of…

The recent SILO-LM paper has a slightly different approach: rather than using input embeddings and prompting the LLM with documents, it searches the database according to the LLM's output embedding and uses KNN search to skew the output embedding vector before token generation. Done that way round, using LLM embeddings outperforms RAG, allegedly. They did it with a custom language model. I really want to give this a…

Interesting! I’ll have to look into that.

Re: LLMs, RAG, and the missing storage layer for AI

#44

Post by BoorishBears »

reissbaker wrote: post #42
BoorishBears wrote: post #39

Earlier quoted context omitted.

You just reinvented a 2 year old technique with a more expensive pipeline and missed performance gains (from the cross-encoder step): https://www.sbert.net/examples/domain_adaptation/README.html https://arxiv.org/abs/2112.07577

I'm aware of more efficient ways to do it! (Hence referencing e.g. doc2query.) But you have to train a model, whereas with an LLM you can get a working version in 5mins of work.

But with even less work you can just pick up a model that was pre-trained using GPL and get great results.

I'm able to pull messy results directly from internet sources and re-rank on the fly with a quantized e5 model small enough to fit in a serverless function.

You don't need a vector database to do all this stuff, people who are paid off people using vector databases are the ones who are hyping them up the most.

Re: LLMs, RAG, and the missing storage layer for AI

#45

Post by amelius »

Unrelated question: is there a standard way for writing down neural network diagrams? I'm thinking of how it is done in electrical circuit schematics, which capture all relevant information in a single diagram, in a (mostly) standardized way.

I've seen the diagrams in DL papers etc. but I guess everyone invents their own conventions, and the diagrams often don't convey the complete flow of information.

Re: LLMs, RAG, and the missing storage layer for AI

#46

Post by viraptor »

brigadier132 wrote: post #15
spott wrote: post #14

Earlier quoted context omitted.

To be fair… semantic similarity isn’t the same as relevance either. They are related, and we frequently assume they are close enough that it doesn’t matter, but they are different.

I disagree, the embeddings are what are used by the llms themselves to produce relevant output and the output is relevant ergo the embeddings do produce relevant output via similarity search

Consider the extreme case: when I ask a question about X, then a page with just the questions about X will get the highest similarity. But what I want in terms of relevance for the answer is a page with a little bit about X and lots of surrounding context that answers the question. By definition the extra context will likely lower the similarity.

Re: LLMs, RAG, and the missing storage layer for AI

#47

Post by gillesjacobs »

amelius wrote: post #45

Unrelated question: is there a standard way for writing down neural network diagrams? I'm thinking of how it is done in electrical circuit schematics, which capture all relevant information in a single diagram, in a (mostly) standardized way. I've seen the diagrams in DL papers etc. but I guess everyone invents their own conventions, and the diagrams often don't convey the complete flow of information.

There are conventions and most libraries have libraries to export diagrams to LaTex or image (e.g., TorchViz).

Visualizations are highly context and usage dependent anyway. Generally, there's is no value in showing fully connected or feed forward layers in detail outside of teaching materials.

Re: LLMs, RAG, and the missing storage layer for AI

#48

Post by amelius »

gillesjacobs wrote: post #47
amelius wrote: post #45

Unrelated question: is there a standard way for writing down neural network diagrams? I'm thinking of how it is done in electrical circuit schematics, which capture all relevant information in a single diagram, in a (mostly) standardized way. I've seen the diagrams in DL papers etc. but I guess everyone invents their own conventions, and the diagrams often don't convey the complete flow of information.

There are conventions and most libraries have libraries to export diagrams to LaTex or image (e.g., TorchViz). Visualizations are highly context and usage dependent anyway. Generally, there's is no value in showing fully connected or feed forward layers in detail outside of teaching materials.

> Generally, there's is no value in showing fully connected or feed forward layers in detail outside of teaching materials.

Well, in electrical circuit diagrams it is customary to draw e.g. a signal bus as a single connection, with the number of wires in the bus written next to it (with a little strike-through line). I'm guessing something similar can be done for DL networks.

Re: LLMs, RAG, and the missing storage layer for AI

#49

Post by zwaps »

godelski wrote: post #38
brigadier132 wrote: post #9

Earlier quoted context omitted.

The vectors are literally constructed so that cosine similarity is semantic similarity. > second unstated assumption is that the vector index can accurately identify the top K vectors by cosine similarity, and that's not true either Its not unstated, its called ANN for a reason

> The vectors are literally constructed so that cosine similarity is semantic similarity. Are they? A learned embedding doesn't guarantee this and a positional embedding certainly doesn't. Our latent embeddings don't either unless you are inferring this through the dot product in the attention mechanism. But that too is learned. There are no guarantees that the similarities that they learn are the same things we cons…

Do you know how embedding models are trained?
Post reply on HN