Ask HN: Have AI learn my own business Knowledge(verity of formats) for chat bot

#1

Post by crazymoka » Tue, Jul 30, 2024, 7:00 PM UTC

Edit. Rephrased the question.

I'm looking for the best way to train from my business knowledge base for openAI, Llama, or Claude on my own private business knowledge base.

That data will then be used in a chatbot(maybe whatsapp) on my website that will either create an appointment for call, send follow up email with more information, or directly connect to an real agent.

I will use it to update my CRM information too.

Knowledge comes in all kinds of formats, PDF, Excel, Power Point Slides, Videos.

Looking for some advice on how to do this on a budget. I am a programmer so I do not mind getting my hands dirty or even running my own server that can do most of the work.

But if there is a 3rd party service or open source tool that does most of this, I'm happy to give it a shot too.

Thanks.

Re: Ask HN: Have AI learn my own business Knowledge(verity of formats) for chat bot

#2

Post by kingkongjaffa » Tue, Jul 30, 2024, 7:35 PM UTC

If you’re on a budget you don’t want to “Train” the model. I.e fine tune.

Since you have multi format data you likely want a pipeline to convert it all to text using various tools, make sure it’s structured and then shove it in a RAG system for the LLM chatbot to work with.

You can get started with lang chain and openAI’s API

Experiment with gpt4o mini for a while to keep costs down and then test if cranking up to gpt4o proper is worth it.

That’s the LLM part solved. You’ll need to control the logic after that depending on controls in your chatbot pop-up window to be able to arrange the calls/send emails etc.

Re: Ask HN: Have AI learn my own business Knowledge(verity of formats) for chat bot

#3

Post by crazymoka » Tue, Jul 30, 2024, 7:50 PM UTC

kingkongjaffa wrote: post #2

If you’re on a budget you don’t want to “Train” the model. I.e fine tune. Since you have multi format data you likely want a pipeline to convert it all to text using various tools, make sure it’s structured and then shove it in a RAG system for the LLM chatbot to work with. You can get started with lang chain and openAI’s API Experiment with gpt4o mini for a while to keep costs down and then test if cranking up to gp…

I can put all the content into text. Would putting it all into json or cvs with headers be helpful? Break down content by topic, categories, links to other pages if it needs to read other content?

Re: Ask HN: Have AI learn my own business Knowledge(verity of formats) for chat bot

#4

Post by abdullin » Tue, Jul 30, 2024, 8:18 PM UTC

I’m consulting multiple teams on shipping LLM-driven business automation. So far I have seen only one case where fine-tuning a model really paid off (and didn’t just blow up the RLHF calibration and caused wild hallucinations).

I would suggest to avoid training and look into RAG systems, prompt engineering and using OpenAI API for a start.

You can do a small PoC quickly using something like LangChain or LlamaIndex. Their pipelines can ingest unstructured data in all file formats, which is good for getting a quick feel.

Afterwards, if you encounter hallucinations in your tasks - throw out vector DB and embeddings into the trashcan (they are pulling junk information into the context and causing hallucinations). Replace embeddings with a RAG based on full text search and query expansion based on the nuances of your business.

If there are any specific types of questions or requests that you need special handling for - add a lightweight router (request classifier) that will direct user request to a dedicated prompt with dedicated data.

By that time you would’ve probably lost all of RAG, replacing it with a couple of prompt templates, a file based knowledge base in markdown and CSV and a few helpers to pull relevant information into the context.

That’s how most of working LLM-driven workflows end up (in my bubble). Maybe just with PostgreSQL and ES instead of file-based knowledge base. But that’s an implementation detail.

Update: if you really want to try fine-tuning your own LLM - this article links to a Google Collab Notebook for the latest Llama 3.1 8B: https://unsloth.ai/blog/llama3-1

It will not learn new things from your data, though. Might just pick up the style.

Re: Ask HN: Have AI learn my own business Knowledge(verity of formats) for chat bot

#5

Post by kingkongjaffa » Tue, Jul 30, 2024, 8:26 PM UTC

crazymoka wrote: post #3

kingkongjaffa wrote: post #2

If you’re on a budget you don’t want to “Train” the model. I.e fine tune. Since you have multi format data you likely want a pipeline to convert it all to text using various tools, make sure it’s structured and then shove it in a RAG system for the LLM chatbot to work with. You can get started with lang chain and openAI’s API Experiment with gpt4o mini for a while to keep costs down and then test if cranking up to gp…

I can put all the content into text. Would putting it all into json or cvs with headers be helpful? Break down content by topic, categories, links to other pages if it needs to read other content?

Yeah whatever will give you accurate vector embeddings.

You’ll combine those json properties into one vector.

For links you might hard code those as an extra part of each object or potentially require another API call to retrieve relevant links from your “link store”.

It’s useful to have benchmarks in place like:

input “should equal” output And start building a suite of test cases to evaluate your set up.

Re: Ask HN: Have AI learn my own business Knowledge(verity of formats) for chat bot

#6

Post by kingkongjaffa » Tue, Jul 30, 2024, 8:29 PM UTC

abdullin wrote: post #4

I’m consulting multiple teams on shipping LLM-driven business automation. So far I have seen only one case where fine-tuning a model really paid off (and didn’t just blow up the RLHF calibration and caused wild hallucinations). I would suggest to avoid training and look into RAG systems, prompt engineering and using OpenAI API for a start. You can do a small PoC quickly using something like LangChain or LlamaIndex. T…

Plus one for this approach, I’m trying to say broadly the same thing with my comment.

What are you using for full text search RAG in production?

Re: Ask HN: Have AI learn my own business Knowledge(verity of formats) for chat bot

#7

Post by abdullin » Tue, Jul 30, 2024, 8:45 PM UTC

kingkongjaffa wrote: post #6

abdullin wrote: post #4

I’m consulting multiple teams on shipping LLM-driven business automation. So far I have seen only one case where fine-tuning a model really paid off (and didn’t just blow up the RLHF calibration and caused wild hallucinations). I would suggest to avoid training and look into RAG systems, prompt engineering and using OpenAI API for a start. You can do a small PoC quickly using something like LangChain or LlamaIndex. T…

Plus one for this approach, I’m trying to say broadly the same thing with my comment. What are you using for full text search RAG in production?

It really depends on the setup that the dev/ops at customer are more comfortable with. Elastic or PostgreSQL can be both fine.

Personally for small cases (e.g. under 50k documents and 20GB of text) I like to use SQLite FTS, while linking text fragments to the additional metadata (native or extracted). This way I can really narrow down the search scope to a few case related documents in each conversation path.

But ultimately the flavor of DB and FTS is just an implementation detail. Most of them will do just fine.

Edit: fixed grammar.

Re: Ask HN: Have AI learn my own business Knowledge(verity of formats) for chat bot

#8

Post by ndr_ » Tue, Jul 30, 2024, 10:13 PM UTC

kingkongjaffa wrote: post #2

If you’re on a budget you don’t want to “Train” the model. I.e fine tune. Since you have multi format data you likely want a pipeline to convert it all to text using various tools, make sure it’s structured and then shove it in a RAG system for the LLM chatbot to work with. You can get started with lang chain and openAI’s API Experiment with gpt4o mini for a while to keep costs down and then test if cranking up to gp…

Depends on the budget, I’d say. If it’s less than 10¢, that may be true. If it‘s 15¢ or more, you could try training with OpenAI: https://ndurner.github.io/training-own-model-finetuning

Re: Ask HN: Have AI learn my own business Knowledge(verity of formats) for chat bot

#9

Post by vannevar » Wed, Jul 31, 2024, 12:42 AM UTC

abdullin wrote: post #4

I’m consulting multiple teams on shipping LLM-driven business automation. So far I have seen only one case where fine-tuning a model really paid off (and didn’t just blow up the RLHF calibration and caused wild hallucinations). I would suggest to avoid training and look into RAG systems, prompt engineering and using OpenAI API for a start. You can do a small PoC quickly using something like LangChain or LlamaIndex. T…

>throw out vector DB and embeddings into the trashcan (they are pulling junk information into the context and causing hallucinations)

Not sure why this would be true. In my experience, semantic search based on a vector index/embeddings pulls in more relevant information than a full-text keyword search. Maybe there is too broad a set of materials in your vector db, or the chunking strategy isn't good?

Re: Ask HN: Have AI learn my own business Knowledge(verity of formats) for chat bot

#10

Post by cranberryturkey » Wed, Jul 31, 2024, 5:46 AM UTC

abdullin wrote: post #4

I’m consulting multiple teams on shipping LLM-driven business automation. So far I have seen only one case where fine-tuning a model really paid off (and didn’t just blow up the RLHF calibration and caused wild hallucinations). I would suggest to avoid training and look into RAG systems, prompt engineering and using OpenAI API for a start. You can do a small PoC quickly using something like LangChain or LlamaIndex. T…

Doesn't RAG approach (and LangChain) both require you send the context data (ie: your book data) in the prompt query api call? How would you fit 20,000 books in that call?