Build a search engine, not a vector DB

I can relate. I have had conversations about enterprise search and how it can help them especially when done with the help of embeddings + LLMs, but many do not see it as a problem. It's a classic case of people you would be selling to have hired analysts for the use case, and do not see it as a prominent problem anymore. Employees would like better search, but not as much that they would go to CTOs and vouch for it.

You can use analogies like:

1. Imagine the world before Google. Web search was a pain. > would be similarly transformative.

2. Every company has an encyclopedia - the guy who knows about the past efforts and is consulted whenever people are trying something new. Search makes that redundant and reduce times.

3. Same with repetitive work because the employees cannot find where the work was done previously.

search is a feature, and unless you address the central pain point that search solves (in terms of revenue), no one will go for it. When you do, you will end up solving the second problem about how leaders never have the issue but employees do.

this is an example: https://platform.openai.com/playground/p/HpFda4ZRXjbbanBwG35...

it's a ReAct loop with search and retrieve action, where I'm simulating the tool by hand. in prod, you'd pick up the output of the Action, run the callback with the LLM input, get the result, and pass the result as 'Observation:' - for the sake of this demo, I'm doing exactly that but manually copy pasting out of wikipedia

works more or less with any backend, and the llm is smart enough to change direction if a search doesn't produce relevant result (and you can see it in the demo). here the loop is cut short because I was running manually, but you can see the important bits.

just implement a retrieve and search function to whatever data source you have, vector or full text, and a couple regex to extract actions and final answer.

pro tip use a expensive llm to run the react loop, and a cheaper llm to summarize articles content after retrieval and before putting it as an observation. ideally you'd want something like "this is a document {document} on this topic: {last_thought}, extract the information relevant to the user question: {question}" trough a cheap llm, so you have the least amount of token into the react loop.

Thanks for the nice words on txtai. There have been times this year I've thought about an alternate 2023 where the focus wasn't LLMs and RAG.

ChatGPT certainly set the tone for the year. Though I will say you haven't heard the last of semantic graphs, semantic paths and some of that work that did happen in late 2022 right before ChatGPT. A bit of a detour? Yes. Perhaps the combination is something that will lead to features even more interesting - time will tell.

if someone has to build this locally to fetch discussion where x topic was discussed or find a person who had shown interest in certain x thing, how does one go about it?

One way of doing it is to embed messages with the added context of previous messages until the topic changes, otherwise, a simple similarity search of user prompt embedding would output messages of irrelevant topics since the context was included from the start.

Then embed the user prompt and perform a similarity search of either the user's query or create a hypothetical statement based on the prompt, also called HyDe approach. You ask an LLM to generate a hypothetical response given the query and then use its vector along with the query vector to enhance search quality.

For example, if the user query is - "find me who is interested in playing Minecraft on Tuesday", the llm will generate a response "I play Minecraft on Tuesdays" and we can search the vector of the llm output in the vector db which is all the messages along with their context.

However, I am not sure how this will work in scenarios where the user has sent a message asking "Will you play Minecraft on Tuesday", and person A has responded with "Yes". how can we have the model find person A? Shall we make a summary of each person based on the conversation with the user?

Also, the whole process might be computationally slow. how do we enhance the speed and performance?

(a noob here who wanted to build a similar solution)

Wow. I learned some stuff about etiquette on HN today.

I'll support you, mnd999. I don't work for a graph dB company. We don't use graph dBs, but I'm considering it. Graph dbs are a legitimate source to feed data I to your RAG system. Our RAG system currently used hybrid search: lexical and semantic. We need to expand our sources, too. I would like to see us use LLMs to rephrase our content (we have a lot of code), and index on that. I think we should build a KG on content quality (we have millions of docs) and software out the things no one likes.

I also think a KG on "learning journeys" would be valuable, but really difficult.