Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Have AI learn my own business Knowledge(verity of formats) for chat bot

news.ycombinator.com

Post reply on HN
Print view View original

11–19 of 19 posts

Re: Ask HN: Have AI learn my own business Knowledge(verity of formats) for chat bot

#11

Post by cranberryturkey »

kingkongjaffa wrote: post #2

If you’re on a budget you don’t want to “Train” the model. I.e fine tune. Since you have multi format data you likely want a pipeline to convert it all to text using various tools, make sure it’s structured and then shove it in a RAG system for the LLM chatbot to work with. You can get started with lang chain and openAI’s API Experiment with gpt4o mini for a while to keep costs down and then test if cranking up to gp…

> shove it in a RAG system for the LLM chatbot to work with

how would I do this with an ollama model?

Re: Ask HN: Have AI learn my own business Knowledge(verity of formats) for chat bot

#12

Post by abdullin »

vannevar wrote: post #9
abdullin wrote: post #4

I’m consulting multiple teams on shipping LLM-driven business automation. So far I have seen only one case where fine-tuning a model really paid off (and didn’t just blow up the RLHF calibration and caused wild hallucinations). I would suggest to avoid training and look into RAG systems, prompt engineering and using OpenAI API for a start. You can do a small PoC quickly using something like LangChain or LlamaIndex. T…

>throw out vector DB and embeddings into the trashcan (they are pulling junk information into the context and causing hallucinations) Not sure why this would be true. In my experience, semantic search based on a vector index/embeddings pulls in more relevant information than a full-text keyword search. Maybe there is too broad a set of materials in your vector db, or the chunking strategy isn't good?

It might depend on the case.

My problem with similarity search - it is unpredictable. It can sometimes miss really obvious matches or pull completely irrelevant snippets. When this happens - this causes downstream hallucinations that are hard to fix.

My customers don’t tolerate hallucinations.

Query expansion with FTS search works more predictably for me. Especially, if we factor in search scope reduction driven by the request classifier (“agent router”)

Re: Ask HN: Have AI learn my own business Knowledge(verity of formats) for chat bot

#13

Post by abdullin »

cranberryturkey wrote: post #10
abdullin wrote: post #4

I’m consulting multiple teams on shipping LLM-driven business automation. So far I have seen only one case where fine-tuning a model really paid off (and didn’t just blow up the RLHF calibration and caused wild hallucinations). I would suggest to avoid training and look into RAG systems, prompt engineering and using OpenAI API for a start. You can do a small PoC quickly using something like LangChain or LlamaIndex. T…

Doesn't RAG approach (and LangChain) both require you send the context data (ie: your book data) in the prompt query api call? How would you fit 20,000 books in that call?

It is impossible to fit all that information into the call.

The whole point of RAG - we (somehow) retrieve only the relevant information and put it into the context to generate the answer.

Re: Ask HN: Have AI learn my own business Knowledge(verity of formats) for chat bot

#14

Post by vannevar »

abdullin wrote: post #12
vannevar wrote: post #9

Earlier quoted context omitted.

>throw out vector DB and embeddings into the trashcan (they are pulling junk information into the context and causing hallucinations) Not sure why this would be true. In my experience, semantic search based on a vector index/embeddings pulls in more relevant information than a full-text keyword search. Maybe there is too broad a set of materials in your vector db, or the chunking strategy isn't good?

It might depend on the case. My problem with similarity search - it is unpredictable. It can sometimes miss really obvious matches or pull completely irrelevant snippets. When this happens - this causes downstream hallucinations that are hard to fix. My customers don’t tolerate hallucinations. Query expansion with FTS search works more predictably for me. Especially, if we factor in search scope reduction driven by t…

For sure it will depend on use case, if you have fairly structured data or a clear domain-specific terminology to rely on, there's probably no reason to use semantic search.

>Query expansion with FTS search works more predictably for me. Especially, if we factor in search scope reduction driven by the request classifier (“agent router”)

You might be able to quantify this and gain some insight into why query expansion/FTS is working better by comparing the precision/recall with a vector db using some set of benchmark docs and queries.

Re: Ask HN: Have AI learn my own business Knowledge(verity of formats) for chat bot

#15

Post by abdullin »

vannevar wrote: post #14
abdullin wrote: post #12

Earlier quoted context omitted.

It might depend on the case. My problem with similarity search - it is unpredictable. It can sometimes miss really obvious matches or pull completely irrelevant snippets. When this happens - this causes downstream hallucinations that are hard to fix. My customers don’t tolerate hallucinations. Query expansion with FTS search works more predictably for me. Especially, if we factor in search scope reduction driven by t…

For sure it will depend on use case, if you have fairly structured data or a clear domain-specific terminology to rely on, there's probably no reason to use semantic search. >Query expansion with FTS search works more predictably for me. Especially, if we factor in search scope reduction driven by the request classifier (“agent router”) You might be able to quantify this and gain some insight into why query expansion…

> For sure it will depend on use case, if you have fairly structured data or a clear domain-specific terminology to rely on

Indeed. This works only in a subset of business domains for me: search and assistants within enterprise knowledge base (e.g. ~40k documents with 20GB of text) within logistics, supply chain, legal, fintech and medtech.

> You might be able to quantify this and gain some insight into why query expansion/FTS is working better by comparing the precision/recall with a vector db using some set of benchmark docs and queries.

Embeddings tend to miss a lot of nuances, plus they are just unpredictable when searching on large sets of text (e.g. 40k documents fragmented), frequently pulling irrelevant texts before the relevant ones. Context contamination leads to hallucinations in our cases.

However with LLM-driven query expansion and FTS search I can get controllable retrieval quality in business tasks. Plus, if something edge case shows up, it is fairly easy to explain and adjust the query expansion logic to cover specific nuances.

This is the setup I'm happy with.

Re: Ask HN: Have AI learn my own business Knowledge(verity of formats) for chat bot

#16

Post by kingkongjaffa »

cranberryturkey wrote: post #11
kingkongjaffa wrote: post #2

If you’re on a budget you don’t want to “Train” the model. I.e fine tune. Since you have multi format data you likely want a pipeline to convert it all to text using various tools, make sure it’s structured and then shove it in a RAG system for the LLM chatbot to work with. You can get started with lang chain and openAI’s API Experiment with gpt4o mini for a while to keep costs down and then test if cranking up to gp…

> shove it in a RAG system for the LLM chatbot to work with how would I do this with an ollama model?

You want to ingest your input documents and generate embeddings using an embedding model like https://ollama.com/blog/embedding-models

Re: Ask HN: Have AI learn my own business Knowledge(verity of formats) for chat bot

#17

Post by pjb88 »

Not sure if this will help, but a while ago I was thoroughly confused about all the AI options (and advice from other people) so spent a while experimenting, now make systems for commercial sometimes, but for a basic-yet-functional knowledge base, that you can expand with whatever tooling you want:

- Don't use llamaindex/llangchain etc. - fine to get started quick but you'll quickly get frustrated when you try to do something different

- Suck in all your files using public libraries. convert to text. Remove obvious crap like line breaks etc. Don't worry about it too much.

- Use postgres as vectorDB - cheap.

- OpenAI is fine, and the docs are great - gpt 3.5 gives fine results; cheapest embedding model fine.

- Spend some time optimising the prompts - that's the most important thing.

I wrote up basics for my specific niche here, has cost/time breakdowns and costs about $4 per month for hosting (and only then because I couldn't face setting up postgres on my other server) and https://superstarsoftware.co.uk/ai-for-drilling-engineers/

(as in: dirt cheap).

I basically made it as a showcase for potential customers, was half thinking of open sourcing it so people can get up and running quickly including with decent frontend, but not sure if there's much appetite since it's basic.

Re: Ask HN: Have AI learn my own business Knowledge(verity of formats) for chat bot

#18

Post by theolivenbaum »

If you want to try our software (includes search, RAG, file handling, and a bunch of integrations, and can be deployed on prem or cloud), happy to give you a license in exchange for some feedback! Just shout me an email at rafael (at) curiosity.ai
Post reply on HN