This is from May 2025, according to the arxiv watermark. Maybe that should be mentioned in the title.
Zebra-Llama – Towards efficient hybrid models
31–40 of 66 posts
Re: Zebra-Llama – Towards efficient hybrid models
#32Looks like the trillions of dollars spent on datacentres will end up being regretted.
Re: Zebra-Llama – Towards efficient hybrid models
#33Earlier quoted context omitted.
Like this? https://huggingface.co/amd/Zebra-Llama-8B-8MLA-24Mamba-SFT
Or like this: https://api-docs.deepseek.com/news/news251201 I don't know what's so special about this paper. - They claim to use MLA to reduce KV cache by 90%. Yeah, Deepseek invented that for Deepseek V2 (and also V3 and Deepseek R1 etc) - They claim to use a hybrid linear attention architecture. So does Deepseek V3.2 and that was weeks ago. Or Granite 4, if you want to go even further back. Or Kimi Linear. Or Qwen3…
From your link: DeepSeek-V3.2 Release 2025/12/01
From Zebra-Llama's arXiv page: Submitted on 22 May 2025
Re: Zebra-Llama – Towards efficient hybrid models
#34Earlier quoted context omitted.
I don't doubt the increase in efficiency. I doubt the "drastically". We already see models become more and more capable per weight and per unit of compute. I don't expect a state-change breakthrough. I expect: more of the same. A SOTA 30B model from 2026 is going to be ~30% better than one from 2025. Now, expecting that to hurt Nvidia? Delusional. No one is going to stop and say "oh wow, we got more inference efficie…
Eh. Right now, Claude is good enough. If LLM development hit a magical wall and never got any better, Claude is good enough to be terrifically useful and there's diminishing returns on how much good we get out of it being at $benchmark. Saying we're satisfied with that... well how many years until efficiency gains from one side and consumer hardware from the other meet in the middle so "good enough for everybody" ope…
Coding capability in and of itself may be "good enough" or close to it, but there's a long way to go before AI can build and operate a product end-to-end. In fairness, a lot of the gap may be tooling.
But the end state in my mind is telling an AI "build me XYZ", having it ask all the important questions over the course of a 30-minute chat while making reasonable decisions on all lower-level issues, then waking up the next morning to a live cloud-hosted test environment at a subdomain of the domain it said it would buy along with test builds of native apps for Android, iOS, Linux, macOS, and Windows, all with near-100% automated test coverage and passing tests. Coding agents feel like magic, but we're clearly not there yet.
And that's just coding. If someone wanted to generate a high-quality custom feature-length movie within the usage limits of a $20/mo AI plan, they'd be sorely disappointed.
Re: Zebra-Llama – Towards efficient hybrid models
#35Earlier quoted context omitted.
Like this? https://huggingface.co/amd/Zebra-Llama-8B-8MLA-24Mamba-SFT
Or like this: https://api-docs.deepseek.com/news/news251201 I don't know what's so special about this paper. - They claim to use MLA to reduce KV cache by 90%. Yeah, Deepseek invented that for Deepseek V2 (and also V3 and Deepseek R1 etc) - They claim to use a hybrid linear attention architecture. So does Deepseek V3.2 and that was weeks ago. Or Granite 4, if you want to go even further back. Or Kimi Linear. Or Qwen3…
"Deepseek hasn't done a full $5.6mil full "
Don't forget the billion dollars or so of GPU's they had access to that they left out of that accounting. Also, the R&D cost of the Meta model they originally used. Then, they added $5.6 million on top of that.
Re: Zebra-Llama – Towards efficient hybrid models
#36Earlier quoted context omitted.
Like this? https://huggingface.co/amd/Zebra-Llama-8B-8MLA-24Mamba-SFT
Or like this: https://api-docs.deepseek.com/news/news251201 I don't know what's so special about this paper. - They claim to use MLA to reduce KV cache by 90%. Yeah, Deepseek invented that for Deepseek V2 (and also V3 and Deepseek R1 etc) - They claim to use a hybrid linear attention architecture. So does Deepseek V3.2 and that was weeks ago. Or Granite 4, if you want to go even further back. Or Kimi Linear. Or Qwen3…
Here's what's important about this paper. It is written by AMD researchers. It shows AMD is investing in AI research. Is this the same level of achievement as DeepSeek 3.2. Most likely not. Do they have novel ideas? Difficult to say, there are hundreds of new ideas being tried in this space. Is this worthless? Most certainly not. In order to make progress in this domain (as in any other), you first need to get your feet wet. You need to play with the various components, and see how they fit together. The idea in this paper is that you can combine somehow SSMs (like Mamba) and LLMs (like LLama). The examples they give are absolute toys compared to DeepSeek 3.2 (the largest is 8 billion parameters, while DeepSeek 3.2 has 671 billion parameters). The comparison you are trying to make simply does not apply. The good news for all of us is that AMD is working in this space.
Re: Zebra-Llama – Towards efficient hybrid models
#37Looks like the trillions of dollars spent on datacentres will end up being regretted.
I should have been an electrician.
Re: Zebra-Llama – Towards efficient hybrid models
#38Earlier quoted context omitted.
Kimi K2 also uses MLA, and Kimi Linear runs Kimi Delta Attention (it's SSM-like) for three out of every four layers (the fourth uses MLA).
Kimi K2 is literally a "copy Deepseek's homework" model. Seriously. It's even exactly 61 layers, the same as Deepseek V3/R1.
For a "copy Deepseek's homework" model, it's really good, preferable to DeepSeek for me (at least prior to V3.2, which I haven't been able to fully put through its paces yet). post-training really makes that much of a difference I guess
Re: Zebra-Llama – Towards efficient hybrid models
#39Earlier quoted context omitted.
Good enough? There's no such thing. People said that "good enough" about GPT-4. Now you say that about Claude Opus 4.5. How long before the treadmill turns, and the very same Opus 4.5 becomes "the bare minimum" - the least capable AI you would actually consider using for simple and unimportant tasks? We have miles and miles of AI advancements ahead of us. The end of that road isn't "good enough". It's "too powerful t…
Elon will boil the oceans if it means not having to deal with poor people.
[dead]
Re: Zebra-Llama – Towards efficient hybrid models
#40Earlier quoted context omitted.
Eh. Right now, Claude is good enough. If LLM development hit a magical wall and never got any better, Claude is good enough to be terrifically useful and there's diminishing returns on how much good we get out of it being at $benchmark. Saying we're satisfied with that... well how many years until efficiency gains from one side and consumer hardware from the other meet in the middle so "good enough for everybody" ope…
Coding capability in and of itself may be "good enough" or close to it, but there's a long way to go before AI can build and operate a product end-to-end. In fairness, a lot of the gap may be tooling. But the end state in my mind is telling an AI "build me XYZ", having it ask all the important questions over the course of a 30-minute chat while making reasonable decisions on all lower-level issues, then waking up the…
>But the end state in my mind is telling an AI "build me XYZ", having it ask all the important questions over the course of a 30-minute chat while making reasonable decisions on all lower-level issues, then waking up the next morning to a live cloud-hosted test environment at a subdomain of the domain it said it would buy along with test builds of native apps for Android, iOS, Linux, macOS, and Windows, all with near-100% automated test coverage and passing tests. Coding agents feel like magic, but we're clearly not there yet.
I'm pretty sure we're there. I'm not sure how interested I am in completely closing that loop and completely removing the human from the loop. But I'm also pretty confident that I could do it with nothing but existing models and software built around them.