Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

news.ycombinator.com

Post reply on HN
Print view View original

201–210 of 247 posts

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#201

Post by seldo »

treprinum wrote: post #156
ilaksh wrote: post #5

You don't train on documents. There are many startups claiming that but they are deliberately using a misleading term because they know that's what people are searching for. You still do RAG. Llamaindex is still the best option that I know of. Most of the startups that have working products are likely using llamaindex. All of the ones that say they are training on documents are actually using RAG. Test it out. If it…

LlamaIndex can't do chunk-level metadata, only document-level metadata, so you can't put precise references to where materials the LLM synthesized answers from originated, e.g. HTML anchors. Just write your own RAG with Pinecone and OpenAI APIs directly.

LlamaIndex lets you attach metadata to Nodes which are basically chunks, although that fact is poorly documented! Will fix.

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#202

Post by moltar »

ilaksh wrote: post #5

You don't train on documents. There are many startups claiming that but they are deliberately using a misleading term because they know that's what people are searching for. You still do RAG. Llamaindex is still the best option that I know of. Most of the startups that have working products are likely using llamaindex. All of the ones that say they are training on documents are actually using RAG. Test it out. If it…

Another super easy option for RAG is AWS Bedrock Knowledge Base. It can ingest docs from S3. Just don’t use the OpenSearch serverless store it’s $$$. Can use a low end RDS with pgvector extension.

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#203

Post by moltar »

haolez wrote: post #142
benjaminwootton wrote: post #3

AWS Bedrock is fairly easy. You can do it in 5 or 6 clicks. You have to upload your documents to S3, create a “Knowledge Base” then sync your documents into a vector database like OpenSearch or PineCone. You are then good to go via their playground or the AWS API. I made a video here describing the process, check around 14 minutes in: https://ensembleanalytics.io/blog/introducing-bedrock-knowle... Bedrock is a decent…

Bedrock is cool, but I found it prohibitively expensive for hobbyists and small companies. On a first glance, it would cost me something like $ 5,000 per month for a simple trained model.

Don’t use OpenSearch it’s way overpriced. Use a cheap-o RDS with pgvector. Everything else is charged like any other LLM by token use.

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#204

Post by ohthehugemanate »

NGL I think this one has passed the point on the tech maturityc curve where it makes sense to roll your own I played with MS office's copilot builder the other day and it's amazing. Point it at a set of base URLs, uploaded files, public or behind authentication. In literal seconds you have a copilot that can be embedded anywhere, including messengers. I gave it the root of Azure documentation, the root of red hat documentation? And the root of the ansible documentation and it's excellent. It uses MS's open source LLM copilot framework, and you can swap out models for an open source one (instead of GPT) if you like.

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#205

Post by pushfoo »

novaRom wrote: post #73

We have to add LLMs and MMMs (multi modal models) into all standard Linux distributions. A service will index all local files creating embedding connectors, this will be used to augment user prompts, and voila we can search for anything with natural language.

> We have to add LLMs and MMMs (multi modal models) into all standard Linux distributions.

Why? This seems like a quick road to remote-execution-vuln-as-system-service.

> and voila we can search for anything with natural language.

What if your model keeps misunderstanding you?

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#206

Post by vl »

JacobiX wrote: post #155

I have a question for which I haven't found a definitive answer yet: is how can one effectively manage typos and Out-of-Vocabulary (OOV) words in RAG systems? For instance, if I search for a specific product name but accidentally mistype it, the resulting encoded vector might not be similar to the vector for the correctly spelled product name ?

AFAIK, modern tokenizers/vocabs don’t have OOV anymore because they use sub-word tokens if there is no entry for the given word. This also works for languages with composite words.

In your example sum of embeddings for non-typo and typo version should have close cosine distance if it trained correctly.

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#207

Post by treprinum »

seldo wrote: post #201
treprinum wrote: post #156

Earlier quoted context omitted.

LlamaIndex can't do chunk-level metadata, only document-level metadata, so you can't put precise references to where materials the LLM synthesized answers from originated, e.g. HTML anchors. Just write your own RAG with Pinecone and OpenAI APIs directly.

LlamaIndex lets you attach metadata to Nodes which are basically chunks, although that fact is poorly documented! Will fix.

Thanks! Even with a better documentation, document importers don't extract node metadata so one needs to write their own "text and metadata extractor" as well. It's then easier to skip LlamaIndex altogether, or just get inspiration from some re-ranking etc. you guys did.

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#208

Post by noiv »

dbish wrote: post #138
noiv wrote: post #134

How you do RAG with embeddings NOT in English? I mean there are a few thousand more languages.

Just use your embeddings model of choice that works with your language, I believe Ada from openai is multilingual but I don’t know what languages it works well on, there are many embedding models out there, huggingface is your friend in this search. The output is just a vector and the rest of the system can basically stay the same. The only other thing that may need to change depending on language is any text preproc…

Well, thanks, how do they deal with technical terms never seen during training?

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#209

Post by averageRoyalty »

wahnfrieden wrote: post #159
d4rkp4ttern wrote: post #85

Earlier quoted context omitted.

Echoing others’ sentiments, I was frustrated with the bloat and obscurity of existing tools. This led me to start building Langroid with an agent-oriented paradigm 8 months ago https://github.com/langroid/langroid We have companies using it in production for various use-cases. They especially like our RAG and multi-agent orchestration. See my other comment for details.

what's the "groid"? isn't that a slur?

If you think that's bad, you're gonna hate Scunthorpe.

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#210

Post by throwup238 »

rolisz wrote: post #161
throwup238 wrote: post #152

Earlier quoted context omitted.

"Toy" is the wrong word to describe it but it seems like another order of magnitude or two increase in context size will solve all their problems. On the other hand I've got a terabyte of text extracted from LibGen - let's say I can ignore the half that is fiction and I can dedupe the rest further by 80% - that's still 100gb. On top of that I've got 300gb of text extracted from court documents and that's just from Ca…

What are you trying to achieve with that dataset from LibGen? I kinda expect that GPT4 was trained on the data that is available on LibGen

I need it to be able to cite answers, explore surrounding context, and while it might have been trained on Libgen, it doesn't mean it "internalized" all the data, let alone enough to be useful.
Post reply on HN