Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 247 posts

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#31

Post by simonw »

dfhg wrote: post #10
ilaksh wrote: post #5

You don't train on documents. There are many startups claiming that but they are deliberately using a misleading term because they know that's what people are searching for. You still do RAG. Llamaindex is still the best option that I know of. Most of the startups that have working products are likely using llamaindex. All of the ones that say they are training on documents are actually using RAG. Test it out. If it…

Another question, which one is preferred, LlamaIndex or Langchain, for RAG? Thanks in advance for your insights.

LlamaIndex is mainly focused on RAG. LangChain does a ton of other stuff too. I'd focus on LlamaIndex first.

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#32

Post by simonw »

kmkarim wrote: post #17

Slightly off topic but is there recommended advice on how to tune / train not for document retrieval but for consistent JSON output with specific enums? i.e given a text, always return back a certain set of fields. For some keys here is the possible set of enums etc. One shot prompting does work but curious how others approach this if you have training data on hand.

For OpenAI, use their functions schema mechanism.

Aside from that, take a look at llama.cpp grammars.

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#33

Post by simonw »

dfhg wrote: post #11
galacticaactual wrote: post #6

Train on your own documents or analyze your own documents for answers? Very different things. For the first (fine tuning) follow “AI Jason” on YouTube. He has some great tutorials. For the second (RAG or similar), fire up a cloud VM with GPUs or use Ollama locally and read through the LlamaIndex docs on how to build a RAG pipeline.

Would you kindly elaborate a little bit the difference between training on own documents vs analyzing documents for answers?

The word "training" implies creating a new model by fine-tuning an existing model on top of new documents.

As several other comments in this thread have already indicated: this is almost always the wrong direction. Which is confusing because it's the direction everyone always assumes they should go in at first.

The approaches that does work is surprisingly simple: take the user's question, search for snippets of your documents that appear to be about that question, then paste all of those snippets into the prompt along with the user's question and see what answer you get.

This is known as RAG: Retrieval Augmented Generation. It's a very powerful approach.

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#34

Post by rjzzleep »

seedless-sensat wrote: post #15
ilaksh wrote: post #5

You don't train on documents. There are many startups claiming that but they are deliberately using a misleading term because they know that's what people are searching for. You still do RAG. Llamaindex is still the best option that I know of. Most of the startups that have working products are likely using llamaindex. All of the ones that say they are training on documents are actually using RAG. Test it out. If it…

What is RAG? That's hard to search for

This one seems like a good summary

Retrieval-Augmented Generation for Large Language Models: A Survey

https://arxiv.org/abs/2312.10997

The photos of this post are also good for a high level look

https://twitter.com/dotey/status/1738400607336120573/photo/2

From the various posts I have seen people claim that phi-2 is a good model to start off from.

If you just want to do embeddings, there are various tutorials to use pgvector for that.

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#36

Post by [deleted] »

hrdwdmrbl wrote: post #23
8organicbits wrote: post #22

Earlier quoted context omitted.

https://docs.aws.amazon.com/bedrock/latest/userguide/quotas....

I’m sorry, I don’t understand those limits. It uses a lot of unfamiliar terms like “batch inference” and “modality”. I just want a nice UI that I can give my hard-drive to and then ask it questions.

[deleted]

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#37

Post by iAkashPaul »

A go-to method is to ingest different chunksizes based on the document hierarchy & then use langchain with a bunch of retrievers depending on the doc type.

Then create an index about the metadata of each doc. So that you can ask the RAGbot what all it can answer about.

Another way to ensure it stays on-domain is to generate synthetic questions & check for similarity against user queries. There's a whole rabbit hole of query decomposition to avoid straying off topic as well.

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#38

Post by ilaksh »

benjaminwootton wrote: post #13
ilaksh wrote: post #5

You don't train on documents. There are many startups claiming that but they are deliberately using a misleading term because they know that's what people are searching for. You still do RAG. Llamaindex is still the best option that I know of. Most of the startups that have working products are likely using llamaindex. All of the ones that say they are training on documents are actually using RAG. Test it out. If it…

To sing the praises of Bedrock again, it does have continuous pre-training as well as RAG “knowledge bases”. The former is based on JSON fragments and the RAG stuff is PDFs and other document formats. With regards to its efficacy, I haven’t gone to production with it yet but I was reasonably impressed. I uploaded 100 legal case documents to Bedrock via Claude and could push it pretty hard asking about the various cas…

I forgot about the continuous pre-training thing. How long and how much did they cost on Bedrock?

I had tried to suggest continuous pre-training to my client but it seemed expensive and when I mentioned that he lost interest and just kept wanting me to do fine tuning.

Also to clarify, did you do the continuous pre-training or RAG? And did you compare the efficacy of one or the other or both?

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#39

Post by benjaminwootton »

ilaksh wrote: post #38
benjaminwootton wrote: post #13

Earlier quoted context omitted.

To sing the praises of Bedrock again, it does have continuous pre-training as well as RAG “knowledge bases”. The former is based on JSON fragments and the RAG stuff is PDFs and other document formats. With regards to its efficacy, I haven’t gone to production with it yet but I was reasonably impressed. I uploaded 100 legal case documents to Bedrock via Claude and could push it pretty hard asking about the various cas…

I forgot about the continuous pre-training thing. How long and how much did they cost on Bedrock? I had tried to suggest continuous pre-training to my client but it seemed expensive and when I mentioned that he lost interest and just kept wanting me to do fine tuning. Also to clarify, did you do the continuous pre-training or RAG? And did you compare the efficacy of one or the other or both?

I used the RAG knowledge bases for most of my testing described above.

I got a toy demo up and running with continuous pre-training but haven’t evaluated it unfortunately.

Post reply on HN