Differential Transformer
arxiv.org
Differential Transformer
1–10 of 188 posts
Re: Differential Transformer
#2Did this mean they solved the hallucination problem of transformers?
edit: not fully but it gives promising results. quiet an improvement actually.
Re: Differential Transformer
#3Very interesting. Currently working on timeseries with Transformers. Let me know if anyone else out there is also reading it from that context.
Re: Differential Transformer
#4Did this mean they solved the hallucination problem of transformers? edit: not fully but it gives promising results. quiet an improvement actually.
The paper says that they've improved hallucination mitigation, but not really "solved" the issue.
Re: Differential Transformer
#5The visualization reveals that Transformer tends to allocate only a small proportion of attention scores to the correct answer, while disproportionately focusing on irrelevant context.
[...] Specifically, we partition the query and key vectors into two groups and compute two separate softmax attention maps. Then the result of subtracting these two maps is regarded as attention scores.
[...] The approach is analogous to noise-canceling headphones and differential amplifiers in electrical engineering, where the difference between two signals cancels out common-mode noise.
Simple change, with seemingly decent improvements across the board.
Re: Differential Transformer
#6Did this mean they solved the hallucination problem of transformers? edit: not fully but it gives promising results. quiet an improvement actually.
that would be huge!
Re: Differential Transformer
#7Like most things in this new world of Machine Learning, I'm really confused why this works?
The analogy to noise-cancelling headphones is helpful but in that case we clearly know which is signal and which is noise. Here, if we knew why would we even bother to the noise-cancelling work?
Re: Differential Transformer
#8The modification is simple and beautiful. And the improvements are quite significant.
Re: Differential Transformer
#9Did this mean they solved the hallucination problem of transformers? edit: not fully but it gives promising results. quiet an improvement actually.
The paper says that they've improved hallucination mitigation, but not really "solved" the issue.
"Hallucination" isn't really a problem that can be "fixed". Its just model error.
The root problem is simply that the model doesn't capture reality, just an approximation. What we are incorrectly calling "hallucination" is just the best the model has to offer.
Re: Differential Transformer
#10The visualization reveals that Transformer tends to allocate only a small proportion of attention scores to the correct answer, while disproportionately focusing on irrelevant context. [...] Specifically, we partition the query and key vectors into two groups and compute two separate softmax attention maps. Then the result of subtracting these two maps is regarded as attention scores. [...] The approach is analogous…
[flagged]