Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs, RAG, and the missing storage layer for AI

blog.lancedb.com

Post reply on HN
Print view View original

11–20 of 64 posts

Re: LLMs, RAG, and the missing storage layer for AI

#11

Post by bugglebeetle »

Jimmc414 wrote: post #8
panarky wrote: post #4

The first unstated assumption is that similar vectors are relevant documents, and for many use cases that's just not true. Cosine similarity != relevance. So if your pipeline pulls 2 or 4 or 12 document chunks into the LLM's context, and half or more of them aren't relevant, does this make the LLM's response more or less relevant? The second unstated assumption is that the vector index can accurately identify the top…

Switching to Word2Vec embeddings led to a substantial improvement in my cosine similarity evaluations for text similarity, but granted I was looking for actual similarity, not relevance. I tried many different methods and had lots of mediocre results initially. code: https://github.com/jimmc414/document_intelligence/blob/main/... https://github.com/jimmc414/document_intelligence

As opposed to sentencebert or what?

Re: LLMs, RAG, and the missing storage layer for AI

#12

Post by antupis »

Charon77 wrote: post #3

A lot of things mentioned are too handwaved and not explained well. It's not explained how vector DB is going to help while incumbents like chatgpt4 can already call functions and do API calls. It doesn't make AI less black box, it's irrelevant and not explained.. There's already existing ways to fine tune models without expensive hardwares such as using LoRA to inject small layers with customized training data, whic…

There is lots of things like which you don’t want leak eg customer specific data. For those cases vectors are great.

Re: LLMs, RAG, and the missing storage layer for AI

#13

Post by Jimmc414 »

bugglebeetle wrote: post #11
Jimmc414 wrote: post #8

Earlier quoted context omitted.

Switching to Word2Vec embeddings led to a substantial improvement in my cosine similarity evaluations for text similarity, but granted I was looking for actual similarity, not relevance. I tried many different methods and had lots of mediocre results initially. code: https://github.com/jimmc414/document_intelligence/blob/main/... https://github.com/jimmc414/document_intelligence

As opposed to sentencebert or what?

DistilBERT and RoBERTa

Re: LLMs, RAG, and the missing storage layer for AI

#14

Post by spott »

brigadier132 wrote: post #9
panarky wrote: post #4

The first unstated assumption is that similar vectors are relevant documents, and for many use cases that's just not true. Cosine similarity != relevance. So if your pipeline pulls 2 or 4 or 12 document chunks into the LLM's context, and half or more of them aren't relevant, does this make the LLM's response more or less relevant? The second unstated assumption is that the vector index can accurately identify the top…

The vectors are literally constructed so that cosine similarity is semantic similarity. > second unstated assumption is that the vector index can accurately identify the top K vectors by cosine similarity, and that's not true either Its not unstated, its called ANN for a reason

To be fair… semantic similarity isn’t the same as relevance either.

They are related, and we frequently assume they are close enough that it doesn’t matter, but they are different.

Re: LLMs, RAG, and the missing storage layer for AI

#15

Post by brigadier132 »

spott wrote: post #14
brigadier132 wrote: post #9

Earlier quoted context omitted.

The vectors are literally constructed so that cosine similarity is semantic similarity. > second unstated assumption is that the vector index can accurately identify the top K vectors by cosine similarity, and that's not true either Its not unstated, its called ANN for a reason

To be fair… semantic similarity isn’t the same as relevance either. They are related, and we frequently assume they are close enough that it doesn’t matter, but they are different.

I disagree, the embeddings are what are used by the llms themselves to produce relevant output and the output is relevant ergo the embeddings do produce relevant output via similarity search

Re: LLMs, RAG, and the missing storage layer for AI

#16

Post by viscanti »

brigadier132 wrote: post #15
spott wrote: post #14

Earlier quoted context omitted.

To be fair… semantic similarity isn’t the same as relevance either. They are related, and we frequently assume they are close enough that it doesn’t matter, but they are different.

I disagree, the embeddings are what are used by the llms themselves to produce relevant output and the output is relevant ergo the embeddings do produce relevant output via similarity search

Not if you're using ANN. In some cases that will be very similar to exhaustive search but in other cases you'll get results that you don't want. You also need embeddings that distribute things mostly evenly across the embedding space (not all will).

Re: LLMs, RAG, and the missing storage layer for AI

#17

Post by spott »

brigadier132 wrote: post #15
spott wrote: post #14

Earlier quoted context omitted.

To be fair… semantic similarity isn’t the same as relevance either. They are related, and we frequently assume they are close enough that it doesn’t matter, but they are different.

I disagree, the embeddings are what are used by the llms themselves to produce relevant output and the output is relevant ergo the embeddings do produce relevant output via similarity search

You probably aren’t using an LLM for your text embeddings for document retrieval (they don’t perform as well as specialist embedding models[0]), and even if they did, you have an embedding about a bare document, without any context of what you are trying to get out of it. If you were to add your context in and then get an embedding, you would get a different answer. As your query gets specific, irrelevant aspects of the embedding space can overwhelm the similarity function, leading to irrelevant answers that are still semantically similar.

[0] https://huggingface.co/spaces/mteb/leaderboard

Re: LLMs, RAG, and the missing storage layer for AI

#18

Post by Nowado »

Jimmc414 wrote: post #8
panarky wrote: post #4

The first unstated assumption is that similar vectors are relevant documents, and for many use cases that's just not true. Cosine similarity != relevance. So if your pipeline pulls 2 or 4 or 12 document chunks into the LLM's context, and half or more of them aren't relevant, does this make the LLM's response more or less relevant? The second unstated assumption is that the vector index can accurately identify the top…

Switching to Word2Vec embeddings led to a substantial improvement in my cosine similarity evaluations for text similarity, but granted I was looking for actual similarity, not relevance. I tried many different methods and had lots of mediocre results initially. code: https://github.com/jimmc414/document_intelligence/blob/main/... https://github.com/jimmc414/document_intelligence

Interesting, do you happen to have some quantitative results on this/additional insights/etc?

I've interpreted transformer vector similarity as 'likelihood to be followed by the same thing' which is close to word2vec's 'sum of likelihoods of all words to be replaced by the other set' (kinda), but also very different in some contexts.

Re: LLMs, RAG, and the missing storage layer for AI

#19

Post by freedmand »

I don’t fully understand the fascination with retrieval augmented generation. The retrieval part is already really good and computationally inexpensive — why not just pass the semantic search results to the user in a pleasant interface and allow them to synthesize their own response? Reading a generated paragraph that obscures the full sourcing seems like a practice that’s been popularized to justify using the shiny new tech, but is the generated part what users actually want? (Not to mention there is no bulletproof way to prevent hallucinations, lies, and prompt injection even with retrieval context.)

Re: LLMs, RAG, and the missing storage layer for AI

#20

Post by jamesblonde »

jarulraj wrote: post #7
jamesblonde wrote: post #2

It's not clear to me that only a vector DB should be used for RAG. Vector DBs give you stochastic responses. For customer chatbots, it seems that structured data - from an operational database or a feature store adds more value. If the user asks about an order they made or a product they have a question about, you use the user-id (when logged in) to retrieve all info about what the user bought recently - the LLM will…

Thanks for sharing that observation on customer chatbots. 1. Will that query look like this: SELECT LLM("{user_question}", order_info) FROM postgres_data.order_table WHERE user_id = “101”; 2. How will a feature store, like Hopsworks, help in this app? Shameless self-plug: We are building EvaDB [1], a query engine for shipping fast AI-powered apps with SQL. Would love to exchange notes on such apps if you're up for it…

Why would your projection be this - SELECT LLM("{user_question}", ?

You can train a small llm on your private data to map the user question to tables in your db.

Then Just select with a limit ( or time bounded). The feature store is just another operational store that could have relevant data for the query.

Post reply on HN