121–130 of 247 posts
Earlier quoted context omitted.
Ask chatgpt next time. "What is rag in context of AI?"
Or just using a traditional search engine and "rag" plus literally any ML/AI/LLM term will yield a half dozen results at the top with "Retrieval-augmented generation" in the page title.
https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
You don't train on documents. There are many startups claiming that but they are deliberately using a misleading term because they know that's what people are searching for. You still do RAG. Llamaindex is still the best option that I know of. Most of the startups that have working products are likely using llamaindex. All of the ones that say they are training on documents are actually using RAG. Test it out. If it…
We just held a workshop about this a few weeks ago: https://red.ht/llmappdev We created a simple chatbot using local models with Ollama (llamacpp), LlamaIndex and streamlit. Have a look at the streamlit folder, it's super easy. I used this simple example to teach about RAG, the importance of the system prompt and prompt injection. The notebook folder has a few more examples, local models can even do natural language…
Earlier quoted context omitted.
The word "training" implies creating a new model by fine-tuning an existing model on top of new documents. As several other comments in this thread have already indicated: this is almost always the wrong direction. Which is confusing because it's the direction everyone always assumes they should go in at first. The approaches that does work is surprisingly simple: take the user's question, search for snippets of your…
> take the user's question, search for snippets of your documents that appear to be about that question, then paste all of those snippets into the prompt along with the user's question and see what answer you get. We use RAG at my job, but we don’t do any preprocessing on the message from the user, so the results are not always great for us. Do any of you have experience using a small local model just for extracting…
Embedding and chunking large amounts of documents is expensive though, in both compute and storage.
The other trick I've been planning to explore is using an LLM to turn the user's question into a small number of normal FTS search queries and then run those to try and get context data.
We have to add LLMs and MMMs (multi modal models) into all standard Linux distributions. A service will index all local files creating embedding connectors, this will be used to augment user prompts, and voila we can search for anything with natural language.
Here's a (video) guide on fine-tuning Mistral 7B with QLoRA: https://www.harpercarroll.com/articles/ai/llm-finetune-own-d... / https://ghostarchive.org/varchive/kmkcNVvEz-k Fine tuning does result in degradation of the overall model ( https://twitter.com/xaiguydotagi/status/1737082280835703142 ) and so various RAG techniques may be desirable. As others have mentioned, LlamaIndex is a neat solution to build RAG pipeli…
I think harpercarrol link is a pretty good one, but it basically just feeds in the documents for completion, which isn't a good approach. The dataset needs to represent how you want to use it.
This one might also be helpful https://www.deeplearning.ai/short-courses/finetuning-large-l...
Honestly surprised how almost everyone is saying to use RAG (on its own). One strong benefit to RAG is the data can change, but has lots of failure modes.
People often use hybrid search (fuzzy or bm25 etc alongside embedding search) which I suppose is still RAG.
But fine-tuning models to be better at RAG is valuable as well, increasing accuracy.
https://ragntune.com/blog/Fine-tuning-an-LLM-to-be-good-at-R...
Ideally, I'd try both. Fine tune on both the documents (create a question / answer dataset with gpt4) and rag instruction fine tune it.
You don't train on documents. There are many startups claiming that but they are deliberately using a misleading term because they know that's what people are searching for. You still do RAG. Llamaindex is still the best option that I know of. Most of the startups that have working products are likely using llamaindex. All of the ones that say they are training on documents are actually using RAG. Test it out. If it…
Earlier quoted context omitted.
We just held a workshop about this a few weeks ago: https://red.ht/llmappdev We created a simple chatbot using local models with Ollama (llamacpp), LlamaIndex and streamlit. Have a look at the streamlit folder, it's super easy. I used this simple example to teach about RAG, the importance of the system prompt and prompt injection. The notebook folder has a few more examples, local models can even do natural language…
looks very promising, do you plan to keep this single repo up to date as new things are released?
I'll probably fix some things this week and then either update it or start from scratch. Guided generation, structured extraction, function calling and multi-modal are things I wanted to add and chainlit looks interesting.
Earlier quoted context omitted.
Would you kindly elaborate a little bit the difference between training on own documents vs analyzing documents for answers?
The word "training" implies creating a new model by fine-tuning an existing model on top of new documents. As several other comments in this thread have already indicated: this is almost always the wrong direction. Which is confusing because it's the direction everyone always assumes they should go in at first. The approaches that does work is surprisingly simple: take the user's question, search for snippets of your…
so is the default to just run the RAG search index on every prompt and if it returns nothing then you get the plain answer from the base model otherwise you get the augmented answer?
Earlier quoted context omitted.
We just held a workshop about this a few weeks ago: https://red.ht/llmappdev We created a simple chatbot using local models with Ollama (llamacpp), LlamaIndex and streamlit. Have a look at the streamlit folder, it's super easy. I used this simple example to teach about RAG, the importance of the system prompt and prompt injection. The notebook folder has a few more examples, local models can even do natural language…
Llamaindex has so mucu potential. Any benchmarks on performance compared to fine-tuning?