Is the architecture they use in this diagram currently the best way to train LLMs in general on custom data sets? https://raw.githubusercontent.com/vanna-ai/vanna/main/img/va... That is, store your trained custom data in vector db and then use RAG to retrieve relevant content and inject that into the prompt of the LLM the user is querying with? As opposed to fine tuning or other methods?
All the podcasts I've been listening to recommend RAG over fine-tuning. My intuition is that having the relevant knowledge in the context rather than the weights brings it closer to the outputs, thereby making it much more likely to provide accurate information and avoid hallucinations/confabulations.
I'm always suspicious that is just because RAG is so much more accessible (both compute wise and in terms of expertise required). There's far more profit in selling something accessible to the masses to a lot of people than something only a niche group of users can do.
I think most people who do actual fine tuning would still probably then use RAG afterwards ...