Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

news.ycombinator.com

Post reply on HN
Print view View original

111–120 of 247 posts

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#111

Post by regularfry »

dfhg wrote: post #8
ilaksh wrote: post #5

You don't train on documents. There are many startups claiming that but they are deliberately using a misleading term because they know that's what people are searching for. You still do RAG. Llamaindex is still the best option that I know of. Most of the startups that have working products are likely using llamaindex. All of the ones that say they are training on documents are actually using RAG. Test it out. If it…

Are there public examples of working products using RAG, compared with fine-tuning or training from scratch?

Amazon Q is (at least partially) a RAG implementation.

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#113

Post by whartung »

tomduncalf wrote: post #61
seedless-sensat wrote: post #15

Earlier quoted context omitted.

What is RAG? That's hard to search for

Retrieval Augmented Generation - in brief, using some kind of search to find relevant documents to the user’s question (often vector DB search, which can search by “meaning”, by also other forms of more traditional search), then injecting those into the prompt to the LLM alongside the question, so it hopefully has facts to refer to (and its “generation” can be “augmented” by documents you’ve “retrieved”, I guess!)

So, as a contrived example, with RAG you make some queries, in some format, like “Who is Sauron?” And then start feeding in what books he’s mentioned in, paragraphs describing him from Tolkien books, things he has done.

Then you start making more specific queries? How old is he, how tall is he, etc.

And the game is you run a “questionnaire AI” that can look at a blob of text, and you ask it “what kind of questions might this paragraph answer”, and then turn around and feed those questions and text back into the system.

Is that a 30,000 foot view really of how this works?

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#115

Post by hrdwdmrbl »

throwup238 wrote: post #109
ankit219 wrote: post #75

I think the answer depends on how many documents you have. To think in terms of tokens (assuming 750-1000 tokens is a page), if you have a good estimate of number of pages you want to query on, you can decide on the approach. Three popular approaches: 1. RAG: Most popular and works really well on smaller datasets. It is limited by number of vectors/embeddings. A typical embedding could be of 1000 tokens in size. Llam…

> I talked to a firm with 70GB worth of data. No way a RAG pipeline would give them results. They are struggling to get LLMs to work for them. Wow so RAG is basically a toy for demos and low effort MVPs. 70GB is tiny, it’d barely qualify as “big data” 20 years ago. Is anyone trying more advanced stuff like knowledge graph augmented generation to try to expand on that?

I have a _small_ e-commerce company and we have >300GB. Most of that bulk is photos and videos though, but in an ideal world I’d like my AI assistant to find that stuff too: “I’m making a Boxing Day ad campaign. Can you show me the ads that we’ve made in previous years and all of the photos that we’ve taken of our new Reindeer and Elf designs?”

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#116

Post by hrdwdmrbl »

xuancanh wrote: post #86
nighthawk454 wrote: post #69

Earlier quoted context omitted.

That’s probably unrealistic at this time

It's doable with Amazon Q, but it's in Preview phase now. https://aws.amazon.com/q/

Is Q the answer to this whole thread? People are talking about AWS Bedrock but that seems like something a startup would build upon and offer something like Q, eh?

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#117

Post by hrdwdmrbl »

bmgoau wrote: post #92

Run https://github.com/imartinez/privateGPT Then make ingest /path/to/folder/with/files Then chat to the LLM. Done. Docs: https://docs.privategpt.dev/overview/welcome/quickstart

Others have said that RAG is only good up to a few 100s of files, eh? And since this is based on IndexLlama which is RAG-based, this would have the same limitation, eh?

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#119

Post by [deleted] »

seedless-sensat wrote: post #15
ilaksh wrote: post #5

You don't train on documents. There are many startups claiming that but they are deliberately using a misleading term because they know that's what people are searching for. You still do RAG. Llamaindex is still the best option that I know of. Most of the startups that have working products are likely using llamaindex. All of the ones that say they are training on documents are actually using RAG. Test it out. If it…

What is RAG? That's hard to search for

[deleted]

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#120

Post by coder543 »

mettamage wrote: post #65
xnx wrote: post #9

GPT-4 Turbo has a 128K (~300 pages) context window, which probably handles a lot of use cases which might have previously needed extra training/refinement.

The chatgtp app says it has a context window of 4096 tokens (gpt 4). How do I get access to turbo?

If you're asking ChatGPT about its own characteristics, you should not believe the responses. Models can't examine themselves, so unless the model was trained on specific information about itself, or unless the info is put into the System Prompt, then it cannot know the answer. A response indicating 4096 tokens would just be a hallucination, or a misunderstanding based on training data that saw references to what older versions of ChatGPT were trained on.

ChatGPT-4 is powered by GPT-4 Turbo at this point, so it has a context window that is much larger than 4096 tokens, whether it knows it or not. The ChatGPT application may limit the context size from reaching the full 128k to keep costs down, and it will be using some of the context window for its own purposes, but it's certainly able to maintain context across more than 4096 tokens of conversation.

Post reply on HN