Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

news.ycombinator.com

Post reply on HN
Print view View original

151–160 of 247 posts

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#151

Post by ZephyrBlu »

lopkeny12ko wrote: post #137
hrdwdmrbl wrote: post #23

Earlier quoted context omitted.

I’m sorry, I don’t understand those limits. It uses a lot of unfamiliar terms like “batch inference” and “modality”. I just want a nice UI that I can give my hard-drive to and then ask it questions.

[flagged]

This attitude puzzles me. "If you wish to make an apple pie from scratch, you must first invent the universe" energy.

We are not always makers. Oftentimes we're consumers as well.

I don't want to read documentation and experiment with my phone, I just want it to work out of the box and do what I expect.

This is standard consumer behaviour and you're lying to yourself if you don't think you act like this with some things.

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#152

Post by throwup238 »

ankit219 wrote: post #133
throwup238 wrote: post #109

Earlier quoted context omitted.

> I talked to a firm with 70GB worth of data. No way a RAG pipeline would give them results. They are struggling to get LLMs to work for them. Wow so RAG is basically a toy for demos and low effort MVPs. 70GB is tiny, it’d barely qualify as “big data” 20 years ago. Is anyone trying more advanced stuff like knowledge graph augmented generation to try to expand on that?

Some caution here. Not everything needs to go into a RAG pipeline (Eg: a database table would not necessarily need to be embedded, but its schema should be.). There would be a lot of repetitions, lots of junk and useless data, and numerical data and parsing through that would be a pain. Then comes how the users would behave. You need a longer string to get accurate results. Most non tech users would rather write shor…

"Toy" is the wrong word to describe it but it seems like another order of magnitude or two increase in context size will solve all their problems.

On the other hand I've got a terabyte of text extracted from LibGen - let's say I can ignore the half that is fiction and I can dedupe the rest further by 80% - that's still 100gb. On top of that I've got 300gb of text extracted from court documents and that's just from California! I haven't even downloaded the federal dump yet, let alone the other 49 states. Even if I limited myself to just the US Code and Federal Code of Regulations, that's hundreds of millions of tokens of very dense text. Embedding based RAG has been pretty much useless in each of these cases but maybe I just suck at implementing the retrieval part.

What's the data size on the GST search engine? Do you have any examples of more complex queries?

The only thing that has even been remotely useful to tackling the kind of questions I want to ask of my data sources is having the LLM generate search queries, navigate the knowledge graph, and rank the relevance of retrieved snippets but that often takes dozens if not hundreds of LLM calls which is incredibly slow and expensive.

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#153

Post by baobun »

_giorgio_ wrote: post #118

If you want a simpler task, like training a mistral llama etc on your documents, to act as a document completer , how would you proceed instead? Probably much easier. Thanks

Markov chains, or, if you need to get fancy, Hidden Markov models?

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#154

Post by kkielhofner »

As others have said you want RAG.

The most feature complete implementation I've seen is h2ogpt[0] (not affiliated).

The code is kind of a mess (most of the logic is in an ~8000 line python file) but it supports ingestion of everything from YouTube videos to docx, pdf, etc - either offline or from the web interface. It uses langchain and a ton of additional open source libraries under the hood. It can run directly on Linux, via docker, or with one-click installers for Mac and Windows.

It has various model hosting implementations built in - transformers, exllama, llama.cpp as well as support for model serving frameworks like vLLM, HF TGI, etc or just OpenAI.

You can also define your preferred embedding model along with various other parameters but I've found the out of box defaults to be pretty sane and usable.

[0] - https://github.com/h2oai/h2ogpt

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#155

Post by JacobiX »

I have a question for which I haven't found a definitive answer yet: is how can one effectively manage typos and Out-of-Vocabulary (OOV) words in RAG systems?

For instance, if I search for a specific product name but accidentally mistype it, the resulting encoded vector might not be similar to the vector for the correctly spelled product name ?

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#156

Post by treprinum »

ilaksh wrote: post #5

You don't train on documents. There are many startups claiming that but they are deliberately using a misleading term because they know that's what people are searching for. You still do RAG. Llamaindex is still the best option that I know of. Most of the startups that have working products are likely using llamaindex. All of the ones that say they are training on documents are actually using RAG. Test it out. If it…

LlamaIndex can't do chunk-level metadata, only document-level metadata, so you can't put precise references to where materials the LLM synthesized answers from originated, e.g. HTML anchors. Just write your own RAG with Pinecone and OpenAI APIs directly.

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#157

Post by akrymski »

simonw wrote: post #123
codetrotter wrote: post #46

Earlier quoted context omitted.

> take the user's question, search for snippets of your documents that appear to be about that question, then paste all of those snippets into the prompt along with the user's question and see what answer you get. We use RAG at my job, but we don’t do any preprocessing on the message from the user, so the results are not always great for us. Do any of you have experience using a small local model just for extracting…

I've been trying out an interesting embedding model that knows how to treat text as a question be as a phrase about the world, and embeds the question such that it's likely to end up close to phrases that might answer that question: https://til.simonwillison.net/llms/embed-paragraphs Embedding and chunking large amounts of documents is expensive though, in both compute and storage. The other trick I've been planning…

> The other trick I've been planning to explore is using an LLM to turn the user's question into a small number of normal FTS search queries and then run those to try and get context data.

I have also been working on this. I still fail to see why this approach isn't the default frankly. There's little benefit to vector databases.

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#159

Post by wahnfrieden »

d4rkp4ttern wrote: post #85
dfhg wrote: post #10

Earlier quoted context omitted.

Another question, which one is preferred, LlamaIndex or Langchain, for RAG? Thanks in advance for your insights.

Echoing others’ sentiments, I was frustrated with the bloat and obscurity of existing tools. This led me to start building Langroid with an agent-oriented paradigm 8 months ago https://github.com/langroid/langroid We have companies using it in production for various use-cases. They especially like our RAG and multi-agent orchestration. See my other comment for details.

what's the "groid"? isn't that a slur?

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#160

Post by viksit »

wahnfrieden wrote: post #159
d4rkp4ttern wrote: post #85

Earlier quoted context omitted.

Echoing others’ sentiments, I was frustrated with the bloat and obscurity of existing tools. This led me to start building Langroid with an agent-oriented paradigm 8 months ago https://github.com/langroid/langroid We have companies using it in production for various use-cases. They especially like our RAG and multi-agent orchestration. See my other comment for details.

what's the "groid"? isn't that a slur?

language android i imagine..
Post reply on HN