Viewing profile — cgel

We have trained a completely attention-free LLM whose performance is competitive with state-of-the-art models. This model, which we call Brumby-14B-Base, has a familiar Transformer…

Thanks for posting! I'm an author of the work. Here to answer any questions.

Yes. That is mostly the idea. But calling the state of a linear transformer KV cache is not quite right. A KV cache grows with the sequence length. But the linear transformer state…

Author. There is a misunderstanding in your argument. Your point is about the dataset being sampled from the true distribution. We are happy with that assumption (it's orthogonal t…

Thanks! We just fixed an issue. I was able to test it on FF for MacOS but it would be very helpful if you could confirm the problem is fixed in your end. Also, I'll admit that mess…

Imagine you have two random variables A and B, which are 0 with prob 0.5 and 1 with prob 0.5. They just have the property that when A=1 then B is always 0 and vice versa. Thus, whe…

Author here. Sorry that the typo and the render errors affected you so much. We don't see any rendering issues on our end, if you tell us what browser you are using maybe we can re…

Author here. The blog post contains a clear explanation of Bayesian Inference and provides careful arguments about its potential benefits and limitations. Further, Bayes rule was t…