What is the hallucination rate of, for example, a Llama3 or GPT4?
They claim 50% with fine-tuning and/or(?) RAG (unclear marketing phrasing imo), and claim their method achieves 5% on the same task which is apparently a text-to-sql task set.
Lamini Memory Tuning: 10x Fewer Hallucinations
11–20 of 63 posts
Re: Lamini Memory Tuning: 10x Fewer Hallucinations
#12[deleted]
Re: Lamini Memory Tuning: 10x Fewer Hallucinations
#13[deleted]
[deleted]
Re: Lamini Memory Tuning: 10x Fewer Hallucinations
#14Woww, very creative and interesting idea: I understand it as: train a bunch of fact-based LoRAs to zero loss (they mention 100k different ones), then use RAG to pick the appropriate Loras for a query. So cool. The only moat I can think of for such a company would be proprietary fact loras- basically licensing a modern ai encyclopedia. Anyway, really nice idea.
I think there is an expert router layer to decide which loras to be integrated at inference time. But they also mention that they freeze the weights for router during training. So it is unclear to me how the router was trained on what loss.
Re: Lamini Memory Tuning: 10x Fewer Hallucinations
#15Woww, very creative and interesting idea: I understand it as: train a bunch of fact-based LoRAs to zero loss (they mention 100k different ones), then use RAG to pick the appropriate Loras for a query. So cool. The only moat I can think of for such a company would be proprietary fact loras- basically licensing a modern ai encyclopedia. Anyway, really nice idea.
I think there is an expert router layer to decide which loras to be integrated at inference time. But they also mention that they freeze the weights for router during training. So it is unclear to me how the router was trained on what loss.
Interesting. That’s kind of surprising to me - it would mean with every new Lora they’d need to fine tune the router, no?
Embedding a description of the Lora and using RAG to pull the nearest Loras in the embedding space is where my mind goes; it’s super extensible, minimal additional training for customer use cases, and the way the Loras probably work it’s not terrible to pull a few extras.
Anyway I just speculate —- no idea what they’re actually doing on the backend.
Re: Lamini Memory Tuning: 10x Fewer Hallucinations
#16The website says:
> At inference time, the model retrieves the most relevant experts at each layer and merges back into the base model to respond to the user query.
The paper says:
> At inference time, only the relevant experts are retrieved from the index, allowing the LLM to store a large number of facts while maintaining low inference latency. We use specialized GPU kernels written in Triton Tillet et al. (2019) to accelerate the lookup of experts.
...but darned if I can understand from either what they're actually doing when they say that.
Why do you need a custom GPU kernel for this outside of the normal NN layers?
Can anyone see an explanation of how they pick which expert to use?
Re: Lamini Memory Tuning: 10x Fewer Hallucinations
#17Am I the only one that cringes at "10x fewer?"
How do I multiply positive numbers and get something smaller?
Is "1/10th" or "90% less" not better arithmetic?
Maybe I should have done more gooder at math but it hurts my ears (eyes).
Re: Lamini Memory Tuning: 10x Fewer Hallucinations
#18Doesn't this make the "AI" even less creative and more like full-text-search instead? What makes some data a "fact"? If everything is written in the training data, in the end, won't everything be treated like a fact? So the LLM will have 100% accuracy and 0% creativity.
Re: Lamini Memory Tuning: 10x Fewer Hallucinations
#19[deleted]
Re: Lamini Memory Tuning: 10x Fewer Hallucinations
#20The website says: > At inference time, the model retrieves the most relevant experts at each layer and merges back into the base model to respond to the user query. The paper says: > At inference time, only the relevant experts are retrieved from the index, allowing the LLM to store a large number of facts while maintaining low inference latency. We use specialized GPU kernels written in Triton Tillet et al. (2019) t…
Agreed, I looked through their “paper” and while it goes through the motions of a scientific paper, there’s barely any reproducible methodology. A single page in their paper, including the diagram.
They do reference some papers I’m not familiar with and say their method is “similar”.
If you check the huggingface page mentioned in a footnote, they have two directories: one for a model, and the other which contains a FAISS index. Although in the paper they say they use cross attention, so I have no idea how those could be combined.