Is there any open source front ends out there? I know of anything LLM but hoping to plug my own home built rag system into a nice front end.
Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?
211–220 of 247 posts
Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?
#212Earlier quoted context omitted.
Bedrock is cool, but I found it prohibitively expensive for hobbyists and small companies. On a first glance, it would cost me something like $ 5,000 per month for a simple trained model.
Don’t use OpenSearch it’s way overpriced. Use a cheap-o RDS with pgvector. Everything else is charged like any other LLM by token use.
This has nothing to do with OpenSearch. Using off the shelf models is cheap and pay as you go. Using custom models is what's expensive.
Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?
#213Easiest is OpenAI assistants api. Use the playground and it’s a no code experience.
How do you upload the documents? Via the API or do you have to upload them beforehand through the UI?
You upload in the GUI. It's available on free accounts as well, as long as you use GPT 3.5. There's a dropdown. It's super-convenient.
I uploaded an AWS study guide, and I'm asking for example questions for my testing. As far as I can see, I can tell no way to determine if it's pulling from the guide, or if it's from GPT's data.
Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?
#214Earlier quoted context omitted.
Retrieval Augmented Generation - in brief, using some kind of search to find relevant documents to the user’s question (often vector DB search, which can search by “meaning”, by also other forms of more traditional search), then injecting those into the prompt to the LLM alongside the question, so it hopefully has facts to refer to (and its “generation” can be “augmented” by documents you’ve “retrieved”, I guess!)
So, as a contrived example, with RAG you make some queries, in some format, like “Who is Sauron?” And then start feeding in what books he’s mentioned in, paragraphs describing him from Tolkien books, things he has done. Then you start making more specific queries? How old is he, how tall is he, etc. And the game is you run a “questionnaire AI” that can look at a blob of text, and you ask it “what kind of questions mi…
Essentially you take any decent model trained on factual information regurgitation, or well any decently well rounded model, a llama 2 variant or something.
Then you craft a prompt for the model along the lines of "you are a helpful assistant, you will provide an answer based on the provided information. If no information matches simply respond with 'I don't know that'".
Then, you take all of your documents and divide them into meaningful chunks, ie by paragraph or something. Then you take these chunks and create embeddings for them. An embedding model is another type (not an llm) that generates vectors for strings of text often based on how similar the words are in _meaning_. Ie if I generate embeddings for the phrase "I have a dog" it might (simplified) be a vector like [0.1,0.2,0.3,0.4]. This vector can be seen as representing a point in a multidimensional space. What an embedding model does with the word meaning is something like if I want to search for "cat" that might embed as a vector [0.42]. Now, say we want to search for the query "which pets do I have" first we generate embeddings for this phrase, the word "pet" might be embedded as [0.41] in the vector. Because it's based on trained meaning, the vectors for "pet" and for "dog" will be close together in our multidimensional space. We can choose how strict we want to be with this search (basically a limit to how close the vectors need to be together in space to count as a match).
Next step is to put this into a vector database, a db designed with vector search operations in mind. We store each chunk, the part of the file it's from and that chunks embedding vector in the database.
Then, when the LLM is queried, say "which pets do I have?", we first generate embeddings for the query, then we use the embedding vector to query our database for things that match close enough in space to be relevant but loose enough that we get "connected" words. This gives us a bunch of our chunks ranked by how close that chunks vector is to our query vector in the multidimensional space. We can then take the n highest ranked chunks, concatenate their original text and prepend this to our original LLM query. The LLM then digests this information and responds in natural language.
So the query sent to the LLM might be something like: "you are a helpful assistant, you will provide an answer based on the provided information. If no information matches simply respond with 'I don't know that'
Information:I have a dog,my dog likes steak,my dog's name is Fenrir
User query: which pets do I have?"
All under "information" is passed in from the chunked text returned from the vector db. And the response from that LLM query would ofc be something like "You have a dog, its name is Fenrir and it likes steak."
Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?
#215I think the answer depends on how many documents you have. To think in terms of tokens (assuming 750-1000 tokens is a page), if you have a good estimate of number of pages you want to query on, you can decide on the approach. Three popular approaches: 1. RAG: Most popular and works really well on smaller datasets. It is limited by number of vectors/embeddings. A typical embedding could be of 1000 tokens in size. Llam…
> I talked to a firm with 70GB worth of data. No way a RAG pipeline would give them results. They are struggling to get LLMs to work for them. Wow so RAG is basically a toy for demos and low effort MVPs. 70GB is tiny, it’d barely qualify as “big data” 20 years ago. Is anyone trying more advanced stuff like knowledge graph augmented generation to try to expand on that?
Right tool for the right job.
With a large amount of data, a large amount of data can be "relevant" with a loose query.
I think in those situations it's fine, using a model with an extra large context and similarity etc filters quite tight.
Developing it to realise when there are too many results and to prompt the user to clarify or be more specific would help.
Companies that want to trawl data like this can just deal with it and pay hardware that can run model for >100k context.
If >allIt's totally doable, but not "out of the box".
Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?
#216Earlier quoted context omitted.
Some caution here. Not everything needs to go into a RAG pipeline (Eg: a database table would not necessarily need to be embedded, but its schema should be.). There would be a lot of repetitions, lots of junk and useless data, and numerical data and parsing through that would be a pain. Then comes how the users would behave. You need a longer string to get accurate results. Most non tech users would rather write shor…
"Toy" is the wrong word to describe it but it seems like another order of magnitude or two increase in context size will solve all their problems. On the other hand I've got a terabyte of text extracted from LibGen - let's say I can ignore the half that is fiction and I can dedupe the rest further by 80% - that's still 100gb. On top of that I've got 300gb of text extracted from court documents and that's just from Ca…
As a human would you read 100GB of data all at once?
Or would you read it bit by bit, taking notes and summarising as you went along. Then compiling your notes/summaries into a final report?
Because I don't see why we expect these models to be so superhuman when a 100K context would already be considered superhuman memory.
Imagine me regurgitating 100k tokens worth of dialogue at you and expecting you to take into account every thing I said. I know I couldn't do it, ha ha.
Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?
#217Earlier quoted context omitted.
Is there a limit? Could I create a knowledge base with 10,000 documents? 100k? 1M?
Even if you could, the problem is that these documents are first chunked into smaller pages, and then embeddings are created. When you ask a question, the algo searches for relevant chunks and passes them to the LLM's overall prompt. If there are too many chunks, or too many chunks with similar content, the search also coupled with the LLM's limited context window would mean only 1-3 chunks get passed. This isn't the…
Easy: if > n chunks returned query "too much information, ask user to clarify".
If you have that much data in your vdb, and the user is querying a slice, it just won't do to have them ask something too generic, they need to be asked to be specific.
Ie "what was that thing from last year?" "Sorry, could you be more specific?" "Oh uh the thing to do with the financial reports" "Financial reports, yes, what about them?" "It was something to do with the way we generate them" "Ah yes, the generation of financial reports; last year in December your team leader said on Slack...etc"
Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?
#218You don't train on documents. There are many startups claiming that but they are deliberately using a misleading term because they know that's what people are searching for. You still do RAG. Llamaindex is still the best option that I know of. Most of the startups that have working products are likely using llamaindex. All of the ones that say they are training on documents are actually using RAG. Test it out. If it…
What’s the benefit of llamaindex over just storing documents in chroma and using chroma to query? I’ve done the latter and trying to understand if there’s a performance gain to the former?
Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?
#219Earlier quoted context omitted.
So, as a contrived example, with RAG you make some queries, in some format, like “Who is Sauron?” And then start feeding in what books he’s mentioned in, paragraphs describing him from Tolkien books, things he has done. Then you start making more specific queries? How old is he, how tall is he, etc. And the game is you run a “questionnaire AI” that can look at a blob of text, and you ask it “what kind of questions mi…
Essentially you take any decent model trained on factual information regurgitation, or well any decently well rounded model, a llama 2 variant or something. Then you craft a prompt for the model along the lines of "you are a helpful assistant, you will provide an answer based on the provided information. If no information matches simply respond with 'I don't know that'". Then, you take all of your documents and divid…
Stupid Question: Eli5; Can/Does/Would it make sense to 'cache' (for lack of a better term) a 'memory' of having answered that question.... and so if that question is asked again, it knows that it has answered it in the past, and can/does better?
(Seems like this is what reinforcement training is, but I am just not sure? Everything seems to mush together when talking about gpts logic)
Re: Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?
#220You can use embedchain[1] to connect various data sources and then get a RAG application running on your local and production very easily. Embedchain is an open source RAG framework and It follows a conventional but configurable approach. The conventional approach is suitable for software engineer where they may not be less familiar with AI. The configurable approach is suitable for ML engineer where they have sophis…
Yes, this is most intuitive and easy to use framework for building RAG applications. Do give it a try!