Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs, RAG, and the missing storage layer for AI

blog.lancedb.com

Post reply on HN
Print view View original

51–60 of 64 posts

Re: LLMs, RAG, and the missing storage layer for AI

#51

Post by tinco »

ianpurton wrote: post #35

As an architect working on LLM applications I have these criteria for a database. - Full SQL support - Has good tooling around migrations (i.e. dbmate) - Good support for running in Kubernetes or in the cloud - Well understood by operations i.e. backups and scaling - Supports vectors and similarity search. - Well supported client libraries So basically Postgres and PgVector.

Can I add one more nice to have? Good support for graph data. I'm not 100% certain on it yet, but there's a lot of ideas surrounding storing knowledge as a graph out there and it makes a lot of intuitive sense. I haven't found a killer use case for it yet as so far I can get by just tagging things and sql querying on the tags is powerful enough.

Maybe someone could pitch in. Is knowledge really a graph (for your problem domain), or is that just some bullshit people made up when they still thought AI could be captured mathematically? It feels to me now knowledge is much more like the way vector embeddings work, it's in a cloud where things are related to each other in an analog or statistical way, not a discrete way.

But, perhaps for similar reasons, vector embeddings haven't been super useful to me in building RAG agents yet. Knowledge is either relevant or it's not, and at least for me if it's relevant it has the keywords or tags I need, and just a straight up SQL query brings it in.

Re: LLMs, RAG, and the missing storage layer for AI

#52

Post by btbuildem »

freedmand wrote: post #19

I don’t fully understand the fascination with retrieval augmented generation. The retrieval part is already really good and computationally inexpensive — why not just pass the semantic search results to the user in a pleasant interface and allow them to synthesize their own response? Reading a generated paragraph that obscures the full sourcing seems like a practice that’s been popularized to justify using the shiny…

For me, the #1 advantage is being able to ask follow-up questions

Re: LLMs, RAG, and the missing storage layer for AI

#53

Post by roseway4 »

ianpurton wrote: post #35

As an architect working on LLM applications I have these criteria for a database. - Full SQL support - Has good tooling around migrations (i.e. dbmate) - Good support for running in Kubernetes or in the cloud - Well understood by operations i.e. backups and scaling - Supports vectors and similarity search. - Well supported client libraries So basically Postgres and PgVector.

You may want to take a look at Zep, an LLM application platform that wraps Postgres, pgvector, embedding models, and more to offer chat memory persistence and document vector search.

The Python and TS SDKs are designed to support drop-in replacements for the bits of LangChain that don’t scale, but nothing stops you accessing Postgres directly.

https://github.com/getzep/zep

Disclosure: I’m the primary author.

Re: LLMs, RAG, and the missing storage layer for AI

#54

Post by rolisz »

godelski wrote: post #38
brigadier132 wrote: post #9

Earlier quoted context omitted.

The vectors are literally constructed so that cosine similarity is semantic similarity. > second unstated assumption is that the vector index can accurately identify the top K vectors by cosine similarity, and that's not true either Its not unstated, its called ANN for a reason

> The vectors are literally constructed so that cosine similarity is semantic similarity. Are they? A learned embedding doesn't guarantee this and a positional embedding certainly doesn't. Our latent embeddings don't either unless you are inferring this through the dot product in the attention mechanism. But that too is learned. There are no guarantees that the similarities that they learn are the same things we cons…

There are embeddings that are trained to reflect similarity, for example SentenceBERT, where the training process pushes pairs of similar sentences (as defined by whoever built the dataset) to have closer embeddings and dissimilar sentences to be further apart.

Re: LLMs, RAG, and the missing storage layer for AI

#55

Post by akgfab »

tinco wrote: post #51
ianpurton wrote: post #35

As an architect working on LLM applications I have these criteria for a database. - Full SQL support - Has good tooling around migrations (i.e. dbmate) - Good support for running in Kubernetes or in the cloud - Well understood by operations i.e. backups and scaling - Supports vectors and similarity search. - Well supported client libraries So basically Postgres and PgVector.

Can I add one more nice to have? Good support for graph data. I'm not 100% certain on it yet, but there's a lot of ideas surrounding storing knowledge as a graph out there and it makes a lot of intuitive sense. I haven't found a killer use case for it yet as so far I can get by just tagging things and sql querying on the tags is powerful enough. Maybe someone could pitch in. Is knowledge really a graph (for your prob…

You can think of a vector database with n vectors as a network whose adjacency matrix is nxn and each edge is represented by whatever similarity metric between nodes you choose to use. So you can have strongly connected edges and weakly connected edges.

Re: LLMs, RAG, and the missing storage layer for AI

#56

Post by J_Shelby_J »

jamesblonde wrote: post #2

It's not clear to me that only a vector DB should be used for RAG. Vector DBs give you stochastic responses. For customer chatbots, it seems that structured data - from an operational database or a feature store adds more value. If the user asks about an order they made or a product they have a question about, you use the user-id (when logged in) to retrieve all info about what the user bought recently - the LLM will…

And for technical documentation or code I'm unclear how well semantic search works for CEQ.

I would assume the embedding model isn't trained on code and specific words that are industry/company specific.

Re: LLMs, RAG, and the missing storage layer for AI

#57

Post by fkyoureadthedoc »

jamesblonde wrote: post #20
jarulraj wrote: post #7

Earlier quoted context omitted.

Thanks for sharing that observation on customer chatbots. 1. Will that query look like this: SELECT LLM("{user_question}", order_info) FROM postgres_data.order_table WHERE user_id = “101”; 2. How will a feature store, like Hopsworks, help in this app? Shameless self-plug: We are building EvaDB [1], a query engine for shipping fast AI-powered apps with SQL. Would love to exchange notes on such apps if you're up for it…

Why would your projection be this - SELECT LLM("{user_question}", ? You can train a small llm on your private data to map the user question to tables in your db. Then Just select with a limit ( or time bounded). The feature store is just another operational store that could have relevant data for the query.

> You can train a small llm on your private data to map the user question to tables in your db.

Can you? You've personally done this? Deployed it to production at some kind of non trivial scale and it's working well? I'm not aware of any "small llm" that approaches the quality of gpt-3.5.

Re: LLMs, RAG, and the missing storage layer for AI

#58

Post by philipodonnell »

jamesblonde wrote: post #20
jarulraj wrote: post #7

Earlier quoted context omitted.

Thanks for sharing that observation on customer chatbots. 1. Will that query look like this: SELECT LLM("{user_question}", order_info) FROM postgres_data.order_table WHERE user_id = “101”; 2. How will a feature store, like Hopsworks, help in this app? Shameless self-plug: We are building EvaDB [1], a query engine for shipping fast AI-powered apps with SQL. Would love to exchange notes on such apps if you're up for it…

Why would your projection be this - SELECT LLM("{user_question}", ? You can train a small llm on your private data to map the user question to tables in your db. Then Just select with a limit ( or time bounded). The feature store is just another operational store that could have relevant data for the query.

This is called Text2SQL or NL2SQL, it’s a surprisingly difficult problem even with RAG and GPT4 as soon as the query is non trivial, especially if there are semantic differences between the question and the db schema.

Re: LLMs, RAG, and the missing storage layer for AI

#59

Post by godelski »

rolisz wrote: post #54
godelski wrote: post #38

Earlier quoted context omitted.

> The vectors are literally constructed so that cosine similarity is semantic similarity. Are they? A learned embedding doesn't guarantee this and a positional embedding certainly doesn't. Our latent embeddings don't either unless you are inferring this through the dot product in the attention mechanism. But that too is learned. There are no guarantees that the similarities that they learn are the same things we cons…

There are embeddings that are trained to reflect similarity, for example SentenceBERT, where the training process pushes pairs of similar sentences (as defined by whoever built the dataset) to have closer embeddings and dissimilar sentences to be further apart.

As the OP points out, Cosine similarity doesn't always equate to relevance. As I was expanding upon, things get really messy as the dimensions increase and your intuition about how vectors relate to one another goes out the window, and fast. Distributional mass is not uniform. Rate of originality increases. And of course, there is no guarantees that latent dimensions align with human meaningful semantic features. There's no pressure to align basis vectors with human perceived semantics. My argument isn't about that there isn't a similarity pressure it's that similarity in high dimensions means different things then similarities in low dimensions. For example, in high dimensions most of a unit cube's mass lies outside the unit sphere, while in 2 or 3 dimensions the unit cube is always contained inside with room to spare. High dimensions are weird and that's what my comment is about because many people are using their lower dimensional intuition for ML.

Re: LLMs, RAG, and the missing storage layer for AI

#60

Post by godelski »

zwaps wrote: post #49
godelski wrote: post #38

Earlier quoted context omitted.

> The vectors are literally constructed so that cosine similarity is semantic similarity. Are they? A learned embedding doesn't guarantee this and a positional embedding certainly doesn't. Our latent embeddings don't either unless you are inferring this through the dot product in the attention mechanism. But that too is learned. There are no guarantees that the similarities that they learn are the same things we cons…

Do you know how embedding models are trained?

Yes. My comment is about the geometry of higher dimensions and their meanings. These are not the same as in {2,3}D
Post reply on HN