Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

news.ycombinator.com

Post reply on HN
Print view View original

171–180 of 247 posts

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#171

Post by ercan »

FergusArgyll wrote: post #81
seedless-sensat wrote: post #15

Earlier quoted context omitted.

What is RAG? That's hard to search for

Off Topic; It fascinates me how much variance there is in peoples searching skills. some people think they are talking to a person when searching e.g 'what is the best way that i can {action}' I think the number one trick is to forget grammar and other language niceties and just enter concepts e.g. 'clean car best'

I found something very annoying while looking for technical data ( a service manual for an ancient medical device - build around 2001).

The same term was the name of the device + something about the power source.

The result from the client network - my phone/client computer nothing related to the search for 4-5 pages.

Same search from work - second result was what I was looking.

So it seems there is a relation with your search history, but somehow connected with the related search history from the same ip/network.

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#172

Post by gsharma »

I have usually seen people recommend to chunk by sentences or paragraphs or some fixed length of characters. IMO, all these are suggested because they are easy to write code for, but in reality, length of a meaningful chunk depends entirely on the data. The way we chunk an FAQ document vs a PRD is different.

Based on this assumption, I have a couple of questions:

1. Is chunking the most significant factor in RAG quality?

2. If there are no limitations, would humans that are experts in that dataset, be the best people to create chunks?

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#173

Post by rmbyrro »

benjaminwootton wrote: post #147
haolez wrote: post #142

Earlier quoted context omitted.

Bedrock is cool, but I found it prohibitively expensive for hobbyists and small companies. On a first glance, it would cost me something like $ 5,000 per month for a simple trained model.

I suspect most of the cost in the OpenSearch vector database? I agree it is high and it is not exactly serverless. You pay by the hour. Bedrock itself is charged based on tokens exchanged. I think Pinecone is a cheaper database for hobby and small business projects, though I haven't looked into it.

No, if you use a custom model (trained with your data), you'll pay around $20 per hour, minimum. That equates to ~$15,000.

You can reduce a lot by committing to a 6-month contract, but it won't get cheaper than about ~$5,000/mo.

That's prohibitively expensive for small projects.

Fine tuning GPT 3.5 is much cheaper.

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#174

Post by aik »

dfhg wrote: post #8
ilaksh wrote: post #5

You don't train on documents. There are many startups claiming that but they are deliberately using a misleading term because they know that's what people are searching for. You still do RAG. Llamaindex is still the best option that I know of. Most of the startups that have working products are likely using llamaindex. All of the ones that say they are training on documents are actually using RAG. Test it out. If it…

Are there public examples of working products using RAG, compared with fine-tuning or training from scratch?

Not public but internally I wrote a tool to help us respond to RFPs. You pass in a question from a new RFP and it outputs surprisingly great answers most of the time. Is writing 75%+ of our RFP responses now (naturally we review and adjust sometimes and as needed). And best of all it was very quickly hacked together and it’s actually useful. Copied questions/answers from all previous ones into a doc, and am using OpenAI embeddings api + FAISS vector db + GPT-4 to load the chunks + store the embeddings + process the resulting chunks.

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#175

Post by rplp »

My approach was not to train the model on the documents, as others mentioned.

I built a vector database from the documents, and I query the questions against it, which is very fast. This is the RAG (retrival augmented generation) step others mentioned.

The results, which are literal extracts from the documents, but short ones, are given to the model, which produces an answer. This is the slow part.

I used many of Langchain's tools to manage the whole process.

You can try it on Supawiki, with one of the featured wikis. Then, if you are ok with a solution that hosts your documents for you, you can upload them and use our solution.

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#176

Post by haolez »

rmbyrro wrote: post #173
benjaminwootton wrote: post #147

Earlier quoted context omitted.

I suspect most of the cost in the OpenSearch vector database? I agree it is high and it is not exactly serverless. You pay by the hour. Bedrock itself is charged based on tokens exchanged. I think Pinecone is a cheaper database for hobby and small business projects, though I haven't looked into it.

No, if you use a custom model (trained with your data), you'll pay around $20 per hour, minimum. That equates to ~$15,000. You can reduce a lot by committing to a 6-month contract, but it won't get cheaper than about ~$5,000/mo. That's prohibitively expensive for small projects. Fine tuning GPT 3.5 is much cheaper.

To be fair, Bedrock is more flexible than OpenAI's fine tuning. It's nice to have at least the option to pay big bucks for this. But it's big bucks nonetheless.

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#178

Post by lmeyerov »

throwup238 wrote: post #109
ankit219 wrote: post #75

I think the answer depends on how many documents you have. To think in terms of tokens (assuming 750-1000 tokens is a page), if you have a good estimate of number of pages you want to query on, you can decide on the approach. Three popular approaches: 1. RAG: Most popular and works really well on smaller datasets. It is limited by number of vectors/embeddings. A typical embedding could be of 1000 tokens in size. Llam…

> I talked to a firm with 70GB worth of data. No way a RAG pipeline would give them results. They are struggling to get LLMs to work for them. Wow so RAG is basically a toy for demos and low effort MVPs. 70GB is tiny, it’d barely qualify as “big data” 20 years ago. Is anyone trying more advanced stuff like knowledge graph augmented generation to try to expand on that?

KG in the intuitive 90's sense is fundamentally worse, but with llm-era rethinking, potentially useful

- kg DBs have the same retrieval scale problem, and often built on top of the same DBs

- traditional kg mining would still use LLMs, except instead of storing rich text ("in a clearly friendly and playful manner, the dog chased the cat") or less-rich high-dimensional chunk embeddings of the same, they get discretized down to much lossier rdf ontologies like (dog,chased,cat). There are uses, like building an entity index, but I wouldn't use for core RAG operations like answering questions accurately

- kg can be useful in the sense of, after chunking & embedding, adding *additional* linkages, such as adding summarization hierarchies and then linking related summaries back to source citation chunks. So runtimes can look up not just similar chunks, but also those that are a logical hop away, even if the embeddings are dissimilar, and via pre-summarization, incorporate a lot more insight. Though that's not a traditional kg, it highlights needing non-vector linkage tracking.

We are working on real-time large-scale projects like teaching LLMs to understand the news as it breaks as part of louie.ai, and if folks have projects like that, happy to chat as we figure out our q1+q2 cohorts. It's a fascinating time -- we had to basically scrap our pre-2023 stack here because of the significant advances, and been amazing being able to tackle much harder problems.

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#179

Post by ParetoOptimal »

viraptor wrote: post #43

So far the recommendations are mostly hosted, so here's one local: https://github.com/weaviate/Verba I'm very happy with its results, even though the system is still young and a little bit janky. You can use it with either GPT API, or your local models through LiteLlm. (I'm running ollama + dolphin-mixtral)

Thanks, I'd rather not feed the borgs.

Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

#180

Post by ParetoOptimal »

ZunarJ5 wrote: post #84

I'm a fan of Khoj. Been using it for months. https://github.com/khoj-ai/khoj

I need to give Khoj a try again. I tried it on my org inbox on a whim, couldn't get useful results, and promptly forgot about it.
Post reply on HN