Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs, RAG, and the missing storage layer for AI

blog.lancedb.com

Post reply on HN
Print view View original

31–40 of 64 posts

Re: LLMs, RAG, and the missing storage layer for AI

#31

Post by petra »

spott wrote: post #14
brigadier132 wrote: post #9

Earlier quoted context omitted.

The vectors are literally constructed so that cosine similarity is semantic similarity. > second unstated assumption is that the vector index can accurately identify the top K vectors by cosine similarity, and that's not true either Its not unstated, its called ANN for a reason

To be fair… semantic similarity isn’t the same as relevance either. They are related, and we frequently assume they are close enough that it doesn’t matter, but they are different.

That's interesting.

Are there any good sources to learn more about that?

Re: LLMs, RAG, and the missing storage layer for AI

#32

Post by reissbaker »

BoorishBears wrote: post #22
brigadier132 wrote: post #9

Earlier quoted context omitted.

The vectors are literally constructed so that cosine similarity is semantic similarity. > second unstated assumption is that the vector index can accurately identify the top K vectors by cosine similarity, and that's not true either Its not unstated, its called ANN for a reason

This is kind of a moot argument, semantic similarity is higher dimensionality than cosine similarity can capture. If I'm using vectors for question/answer, then: "What is a cat" and "What is a dog" Should be more dissimilar than the documents answering either. If I'm using it for FAQ filtering then they should be more similar.

I've had decent results using a doc2query style approach:

    1. Ask an LLM to return a list of questions answered by the document
    2. Store the embeddings of the questions along with a document ID
    3. On user query, get the embedding of the user query
    4. KNN cosine similarity search the user embedding vs. the corpus of question embeddings
    5. Return the highest ranked documents
You can tweak this approach depending on your use case, so that in step 1 you generate embeddings that are more similar to the types of things you want returned in step 5. If you want the answer to "What is a cat" to be similar to "What is a dog," you'd prompt/finetune the LLM in step 1 to generate broad questions that would encompass both; if you want them to be very different, you'd do the opposite and avoid generalities.

Re: LLMs, RAG, and the missing storage layer for AI

#34

Post by regularfry »

spott wrote: post #17
brigadier132 wrote: post #15

Earlier quoted context omitted.

I disagree, the embeddings are what are used by the llms themselves to produce relevant output and the output is relevant ergo the embeddings do produce relevant output via similarity search

You probably aren’t using an LLM for your text embeddings for document retrieval (they don’t perform as well as specialist embedding models[0]), and even if they did, you have an embedding about a bare document, without any context of what you are trying to get out of it. If you were to add your context in and then get an embedding, you would get a different answer. As your query gets specific, irrelevant aspects of…

The recent SILO-LM paper has a slightly different approach: rather than using input embeddings and prompting the LLM with documents, it searches the database according to the LLM's output embedding and uses KNN search to skew the output embedding vector before token generation. Done that way round, using LLM embeddings outperforms RAG, allegedly.

They did it with a custom language model. I really want to give this a try with llama2 embeddings but haven't had the bandwidth yet (and llama2's embedding vectors are inconveniently huge, but that's a different problem).

Re: LLMs, RAG, and the missing storage layer for AI

#35

Post by ianpurton »

As an architect working on LLM applications I have these criteria for a database.

- Full SQL support

- Has good tooling around migrations (i.e. dbmate)

- Good support for running in Kubernetes or in the cloud

- Well understood by operations i.e. backups and scaling

- Supports vectors and similarity search.

- Well supported client libraries

So basically Postgres and PgVector.

Re: LLMs, RAG, and the missing storage layer for AI

#36

Post by nottheengineer »

freedmand wrote: post #19

I don’t fully understand the fascination with retrieval augmented generation. The retrieval part is already really good and computationally inexpensive — why not just pass the semantic search results to the user in a pleasant interface and allow them to synthesize their own response? Reading a generated paragraph that obscures the full sourcing seems like a practice that’s been popularized to justify using the shiny…

I really like using LLMs to learn stuff because they can explain anything at the exact level I need. Hallucination is a big problem with that and RAG pretty much solves it. If I give chatGPT a good stackoverflow post and tell it to dumb it down for me, it does very well. RAG just automates that process with the added benefit of not letting the LLM decide which information to retrieve, which should greatly reduce the chance of accidentally biasing the model with your prompt.

Re: LLMs, RAG, and the missing storage layer for AI

#37

Post by seanhunter »

ianpurton wrote: post #35

As an architect working on LLM applications I have these criteria for a database. - Full SQL support - Has good tooling around migrations (i.e. dbmate) - Good support for running in Kubernetes or in the cloud - Well understood by operations i.e. backups and scaling - Supports vectors and similarity search. - Well supported client libraries So basically Postgres and PgVector.

Exactly. The whole point about databases is you don't need "a database for AI" you need a database, ideally with an extension to add additional AI functionality (ie postgres and pgvector). Trying to take a special store you invent for AI and retrofit all the desirable things you need to make it work properly in the context of a real application you're just going to end up with a mess.

As a thought-experiment for people who don't understand why you need (for example) regular relational columns alongside vector storage, consider how you would implement RAG for a set of documents where not everyone has permission to view every document. In the pgvector case it's easy - I can add one or more label columns and then when I do my search query filter to only include labels that user has permission to view. Then my vector similarity results will definitely not include anything that violates my access control. Trivial with something like pgvector - basically impossible (afaics) with special-purpose vector stores.

Or think about ranking. Say you want to do RAG over a space where you want to prioritise the most recent results, not just pure similarity. Or prioritise on a set of other features somehow (source credibility whatever). Easy to do if you have relational columns, no bueno if you just have a vector store.

And that's not to mention the obvious things around ACID, availability, recovery, replication, etc.

Re: LLMs, RAG, and the missing storage layer for AI

#38

Post by godelski »

brigadier132 wrote: post #9
panarky wrote: post #4

The first unstated assumption is that similar vectors are relevant documents, and for many use cases that's just not true. Cosine similarity != relevance. So if your pipeline pulls 2 or 4 or 12 document chunks into the LLM's context, and half or more of them aren't relevant, does this make the LLM's response more or less relevant? The second unstated assumption is that the vector index can accurately identify the top…

The vectors are literally constructed so that cosine similarity is semantic similarity. > second unstated assumption is that the vector index can accurately identify the top K vectors by cosine similarity, and that's not true either Its not unstated, its called ANN for a reason

> The vectors are literally constructed so that cosine similarity is semantic similarity.

Are they? A learned embedding doesn't guarantee this and a positional embedding certainly doesn't. Our latent embeddings don't either unless you are inferring this through the dot product in the attention mechanism. But that too is learned. There are no guarantees that the similarities that they learn are the same things we consider as similarities. High dimensional space is really weird.

And while we're at it, we should mention that methods like t-SNE and UMAP are clustering algorithms not dimensional reduction. Just because they can find ways to cluster the data in a lower dimensional projection (epic mapping) doesn't mean that they are similar in the higher dimensional space. It all depends on the ability to unknot in the higher dimensional space.

It is extremely important to do what the OP is doing and consider the assumptions of the model, data, and measurements. Good results do not necessarily mean good methods. I like to say that you don't need to know math to make a good model, but you do need to know math to know why your model is wrong. Your comment just comes off as dismissive rather than actually countering the claims. There's plenty more assumptions than OP listed too. But their assumptions don't mean the model won't work, it just means what constraints the model is working under. We want to understand the constraints/assumptions if we want to make better models. Large models have advantages because they can have larger latent spaces and that gives them a lot of freedom to unknot data and move them around as they please. But that doesn't mean the methods are efficient.

Re: LLMs, RAG, and the missing storage layer for AI

#39

Post by BoorishBears »

reissbaker wrote: post #32
BoorishBears wrote: post #22

Earlier quoted context omitted.

This is kind of a moot argument, semantic similarity is higher dimensionality than cosine similarity can capture. If I'm using vectors for question/answer, then: "What is a cat" and "What is a dog" Should be more dissimilar than the documents answering either. If I'm using it for FAQ filtering then they should be more similar.

I've had decent results using a doc2query style approach: 1. Ask an LLM to return a list of questions answered by the document 2. Store the embeddings of the questions along with a document ID 3. On user query, get the embedding of the user query 4. KNN cosine similarity search the user embedding vs. the corpus of question embeddings 5. Return the highest ranked documents You can tweak this approach depending on your…

You just reinvented a 2 year old technique with a more expensive pipeline and missed performance gains (from the cross-encoder step):

https://www.sbert.net/examples/domain_adaptation/README.html https://arxiv.org/abs/2112.07577

Re: LLMs, RAG, and the missing storage layer for AI

#40

Post by jorgemf »

freedmand wrote: post #19

I don’t fully understand the fascination with retrieval augmented generation. The retrieval part is already really good and computationally inexpensive — why not just pass the semantic search results to the user in a pleasant interface and allow them to synthesize their own response? Reading a generated paragraph that obscures the full sourcing seems like a practice that’s been popularized to justify using the shiny…

The main reason is that you might not want the raw information but some reasoning above. LLM is not only the context but all the information it has been trained with. For example a math student is making a question, it doesn't want the raw theorems but some reasoning with them, and currently LLM can do that. It will make mistakes sometimes because of hallucinations, but for not very difficult questions it usually gives you the right answer. And that helps a lot when you are not an expert in the domain. And that is the reason GPT4 is a great tool for students, it helps you to understand the basics as if you have a teacher with you.
Post reply on HN