Since no one has mentioned it so far: I did just this recently with txtai in a few lines of code.
Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?
131–140 of 247 posts
Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?
#132Run https://github.com/imartinez/privateGPT Then make ingest /path/to/folder/with/files Then chat to the LLM. Done. Docs: https://docs.privategpt.dev/overview/welcome/quickstart
I've tried LocalGPT, PrivateGPT, and H2OGPT. Have you been satisfied with the responses you get from PrivateGPT? When I tried it, it seemed very shallow/cursory in its responses. I saw much more detailed and complete responses when trying H2OGPT.
Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?
#133I think the answer depends on how many documents you have. To think in terms of tokens (assuming 750-1000 tokens is a page), if you have a good estimate of number of pages you want to query on, you can decide on the approach. Three popular approaches: 1. RAG: Most popular and works really well on smaller datasets. It is limited by number of vectors/embeddings. A typical embedding could be of 1000 tokens in size. Llam…
> I talked to a firm with 70GB worth of data. No way a RAG pipeline would give them results. They are struggling to get LLMs to work for them. Wow so RAG is basically a toy for demos and low effort MVPs. 70GB is tiny, it’d barely qualify as “big data” 20 years ago. Is anyone trying more advanced stuff like knowledge graph augmented generation to try to expand on that?
Some caution here. Not everything needs to go into a RAG pipeline (Eg: a database table would not necessarily need to be embedded, but its schema should be.). There would be a lot of repetitions, lots of junk and useless data, and numerical data and parsing through that would be a pain. Then comes how the users would behave. You need a longer string to get accurate results. Most non tech users would rather write shorter strings and expect technology to read their mind. (it's a human issue and not tech issue)
A simpler way here is just train the model unsupervised so all the knowledge is there in the model, and instruction tune it on the use-cases you want. Simpler from human effort perspective. Somewhat costly though the cost of storing that many vectors would be more than training the model itself. Everything else requires a lot of custom effort. Knowledge graph augmentation is probably the next step in the hype cycle, but it does not solve the fundamental human problem of writing fewer letters. (Training solves as changing 1-2 keywords do the trick if the generic string does not get the answer. See how Chatgpt changes answers if you tweak your prompt a bit). In a way RAG is an engineering solution to what is basically a data problem. It works for many cases, but when it does not, people will have to solve it via data science.
> Wow so RAG is basically a toy for demos and low effort MVPs
I would not say it's for demos or low effort MVPs. Many companies wont have that amount of data. You can also segregate it by teams. Eg: customer support has one, sales has one, product has one. Then, a golden use case is for parsing user docs. We created one for GST queries in India that works quite well.[1]. It's a search engine, but points to right docs at the source when you ask about any clause. Useful for CAs only and addresses a very narrow use case.(it's a market need as the notifications are published in PDF format and not indexed by Google)
Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?
#134How you do RAG with embeddings NOT in English? I mean there are a few thousand more languages.
Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?
#135Earlier quoted context omitted.
> I talked to a firm with 70GB worth of data. No way a RAG pipeline would give them results. They are struggling to get LLMs to work for them. Wow so RAG is basically a toy for demos and low effort MVPs. 70GB is tiny, it’d barely qualify as “big data” 20 years ago. Is anyone trying more advanced stuff like knowledge graph augmented generation to try to expand on that?
I have a _small_ e-commerce company and we have >300GB. Most of that bulk is photos and videos though, but in an ideal world I’d like my AI assistant to find that stuff too: “I’m making a Boxing Day ad campaign. Can you show me the ads that we’ve made in previous years and all of the photos that we’ve taken of our new Reindeer and Elf designs?”
That can be done if we use Imagebind from meta(embeds text, image, video, audio in same vector space). I would want to explore this if possible just for a POC if you are okay with it. Would you be interested?
Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?
#136how to run a local llm model for RAG apps. Retrieval documents are turkish. But ı would to analyze this documents with llm. But ı have not a turkish local llm. How to solve this problem. Out of fine-tune and training.
If the LLM you use supports Turkish (I am pretty sure chatgpt does) then the language doesn’t matter. Augment the Generation by Retrieving turkish documents / snippets.
Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?
#137Earlier quoted context omitted.
https://docs.aws.amazon.com/bedrock/latest/userguide/quotas....
I’m sorry, I don’t understand those limits. It uses a lot of unfamiliar terms like “batch inference” and “modality”. I just want a nice UI that I can give my hard-drive to and then ask it questions.
[flagged]
Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?
#138How you do RAG with embeddings NOT in English? I mean there are a few thousand more languages.
Just use your embeddings model of choice that works with your language, I believe Ada from openai is multilingual but I don’t know what languages it works well on, there are many embedding models out there, huggingface is your friend in this search. The output is just a vector and the rest of the system can basically stay the same. The only other thing that may need to change depending on language is any text preprocessing you need to do like word or sentence breaking for languages with compound words (German), agglutination (Turkish), etc.
Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?
#139Earlier quoted context omitted.
Or just using a traditional search engine and "rag" plus literally any ML/AI/LLM term will yield a half dozen results at the top with "Retrieval-augmented generation" in the page title.
Or if GGP can't think of an AI-related term they can use HN search. Searching 'rag' shows the term on the first page of results: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
Searching for "RAG" on Kagi and Google give some AI-related results fairly high up, including results that explain it and say what it stands for.
Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?
#140I'm curious about this as well but my data is mostly (95%) numerical metrics. Is there a "RAG" mechanism for numerical data instead of text? My use case is data analysis, insight discovery for example.