Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zebra-Llama – Towards efficient hybrid models

arxiv.org

Post reply on HN
Print view View original

51–60 of 66 posts

Re: Zebra-Llama – Towards efficient hybrid models

#51

Post by Palmik »

T-A wrote: post #33
jychang wrote: post #24

Earlier quoted context omitted.

Or like this: https://api-docs.deepseek.com/news/news251201 I don't know what's so special about this paper. - They claim to use MLA to reduce KV cache by 90%. Yeah, Deepseek invented that for Deepseek V2 (and also V3 and Deepseek R1 etc) - They claim to use a hybrid linear attention architecture. So does Deepseek V3.2 and that was weeks ago. Or Granite 4, if you want to go even further back. Or Kimi Linear. Or Qwen3…

From your link: DeepSeek-V3.2 Release 2025/12/01 From Zebra-Llama's arXiv page: Submitted on 22 May 2025

DeepSeek's MLA paper was published in 2024: https://arxiv.org/abs/2405.04434

DeepSeek's Sparse Attention paper was published in February: https://arxiv.org/abs/2502.11089

DeepSeek 3.2 Exp (combining MLA and DSA) was released in September.

You also had several other Chinese hybrid models, like Qwen3 Next and Minimax M1.

Re: Zebra-Llama – Towards efficient hybrid models

#52

Post by nl »

jychang wrote: post #46
SilverElfin wrote: post #42

Earlier quoted context omitted.

How did you get all this info about how each is trained? Is that something they admit now or is it through leaks?

Deepseek? It's literally in their research papers. OpenAI? The OpenAI head of research @markchen90 straight up admitted it in a podcast. https://x.com/petergostev/status/1995744289079656834 "In the last 2 years we've put so much resourcing into, into reasoning and one byproduct of that is you lose a little bit of muscle on pre training and post training." "In the last six months, @merettm and I have done a lot of wor…

Semi-analysis also believes they haven't done a fill pretraining run since 4o (except for GPT-4.5): https://open.substack.com/pub/semianalysis/p/tpuv7-google-ta...

Re: Zebra-Llama – Towards efficient hybrid models

#53

Post by grogers »

colechristensen wrote: post #30
ACCount37 wrote: post #19

Earlier quoted context omitted.

Good enough? There's no such thing. People said that "good enough" about GPT-4. Now you say that about Claude Opus 4.5. How long before the treadmill turns, and the very same Opus 4.5 becomes "the bare minimum" - the least capable AI you would actually consider using for simple and unimportant tasks? We have miles and miles of AI advancements ahead of us. The end of that road isn't "good enough". It's "too powerful t…

I can build fully functional applications without writing a single line of code with Claude. In my free time. On a weekend. I'm going to release one of them pretty soon. A toddler being able to do this instead of an industry veteran isn't that compelling. Avoiding the few pitfalls of the LLM getting stuck and taking a while to get out isn't that valuable. >Good enough? There's no such thing. This is just wrong. Maybe…

LLM only fairly recently underwent a step change from "maybe someday" to actually useful now. That opened many new doors that people didn't even think were possible. Getting incrementally better at something they are already pretty good at isn't that impressive. But getting drastically better at something they are currently bad at, will drive new models and new research.

Re: Zebra-Llama – Towards efficient hybrid models

#54

Post by nl »

buu700 wrote: post #34
colechristensen wrote: post #15

Earlier quoted context omitted.

Eh. Right now, Claude is good enough. If LLM development hit a magical wall and never got any better, Claude is good enough to be terrifically useful and there's diminishing returns on how much good we get out of it being at $benchmark. Saying we're satisfied with that... well how many years until efficiency gains from one side and consumer hardware from the other meet in the middle so "good enough for everybody" ope…

Coding capability in and of itself may be "good enough" or close to it, but there's a long way to go before AI can build and operate a product end-to-end. In fairness, a lot of the gap may be tooling. But the end state in my mind is telling an AI "build me XYZ", having it ask all the important questions over the course of a 30-minute chat while making reasonable decisions on all lower-level issues, then waking up the…

> But the end state in my mind is telling an AI "build me XYZ", having it ask all the important questions over the course of a 30-minute chat while making reasonable decisions on all lower-level issues, then waking up the next morning to a live cloud-hosted test environment at a subdomain of the domain it said it would buy along with test builds of native apps for Android, iOS, Linux, macOS, and Windows, all with near-100% automated test coverage and passing tests. Coding agents feel like magic, but we're clearly not there yet.

If you take out the native builds we are there now. V0, Lovable, etc really do a great job of this. If you want an IDE-like environment Antigravity is pretty good too.

The native builds thing is completely doable too. I've built cross platform apps in 30 minutes of my time using Codex+Flutter. It really does work.

Re: Zebra-Llama – Towards efficient hybrid models

#55

Post by nl »

buu700 wrote: post #48
colechristensen wrote: post #47

Earlier quoted context omitted.

My information to the contrary is my experience in the last few weeks building things with LLMs including tooling to help build things with LLMs. The is experience is one of ... I'm a product manager and devsecops engineer bullying an LLM with the psychology of a toddler into building great software which it can do very successfully. A single instance of a model with a single rolling context window and one set of pro…

Just to be clear, what I was proposing was a single tool which would, on the basis of a single ~30-minute interaction, purchase a domain name, set up a cloud environment, build a full-stack application + cross-platform native apps + useful tests with near-100% coverage, deploy a live test environment, and compile each platform's native app — all entirely autonomously. Are you saying you've used or built something sim…

Have you tried Loveable, Replit, V0 etc?

Outside of purchasing the domain and native apps for you they cover a very significant amount of this.

If you insist on Native Apps, it's possible Google Jules could do it. With Gemini 2.5 it wasn't strong enough but I think it has Gemini 3 now which can definitely do native apps just fine.

Re: Zebra-Llama – Towards efficient hybrid models

#56

Post by YouAreWRONGtoo »

ACCount37 wrote: post #19
colechristensen wrote: post #15

Earlier quoted context omitted.

Eh. Right now, Claude is good enough. If LLM development hit a magical wall and never got any better, Claude is good enough to be terrifically useful and there's diminishing returns on how much good we get out of it being at $benchmark. Saying we're satisfied with that... well how many years until efficiency gains from one side and consumer hardware from the other meet in the middle so "good enough for everybody" ope…

Good enough? There's no such thing. People said that "good enough" about GPT-4. Now you say that about Claude Opus 4.5. How long before the treadmill turns, and the very same Opus 4.5 becomes "the bare minimum" - the least capable AI you would actually consider using for simple and unimportant tasks? We have miles and miles of AI advancements ahead of us. The end of that road isn't "good enough". It's "too powerful t…

[dead]

Re: Zebra-Llama – Towards efficient hybrid models

#57

Post by cubefox »

jychang wrote: post #17
a_wild_dandan wrote: post #12

If the claims in the abstract are true, then this is legitimately revolutionary. I don’t believe it. There are probably some major constraints/caveats that keep these results from generalizing. I’ll read through the paper carefully this time instead of a skim and come back with thoughts after I’ve digested it.

What's not to believe? Qwerky-32b has already done something similar as a finetune of QwQ-32b but not using traditional attention architecture. And hybrid models aren't new, MLA based hybrid models is basically just Deepseek V3.2 in a nutshell. Note that Deepseek V3.2 (and V3.1, R1, and V3... and V2 actually) all use MLA. Deepseek V3.2 is what adds the linear attention stuff. Actually, since Deepseek V3.1 and Deepsee…

DeepSeek-V3.2 is a sparse attention architecture, while Zebra-Llama is a hybrid attention/SSM architecture. The outcome might be similar in some ways (close to linear complexity) but I think they are otherwise quite different.

Re: Zebra-Llama – Towards efficient hybrid models

#58

Post by buu700 »

nl wrote: post #55
buu700 wrote: post #48

Earlier quoted context omitted.

Just to be clear, what I was proposing was a single tool which would, on the basis of a single ~30-minute interaction, purchase a domain name, set up a cloud environment, build a full-stack application + cross-platform native apps + useful tests with near-100% coverage, deploy a live test environment, and compile each platform's native app — all entirely autonomously. Are you saying you've used or built something sim…

Have you tried Loveable, Replit, V0 etc? Outside of purchasing the domain and native apps for you they cover a very significant amount of this. If you insist on Native Apps, it's possible Google Jules could do it. With Gemini 2.5 it wasn't strong enough but I think it has Gemini 3 now which can definitely do native apps just fine.

Thanks for the recommendations. Regarding your other comment, Flutter is what I've landed on as well for my next cross-platform app project, and I'm currently in the middle of developing a spec for a fairly complex agentic system that I'm going to try having Codex two-shot (basic project setup + file stubs + exhaustive tests -> manual checkpoint -> TDD the rest).

I haven't tried Lovable, V0, or Jules, but I really like Replit for certain things. Having said that, based on my experience, I would characterize it as an amazing tool for rapid frontend iteration with prototype-level backend creation. I'm sure it's gotten better at one-shotting since I tried Agent 2 with Sonnet 3.7 in May, but would still be very (pleasantly) surprised to see that Agent 3 with current models could meet the incredibly high bar of wholly replacing a human dev team.

The fact that tools like Replit also include their own hosting environments is definitely neat, but not really what I was getting at as far as deployment. What I had in mind was managing arbitrary cloud platforms, setting up an optimal architecture for your anticipated scale and usage patterns — whether that's a single Hetzner instance with SQLite or horizontally scaled app servers behind an API gateway with Kafka, Valkey, and Spanner or ScyllaDB — and doing all the DevOps to handle that along with things like CI/CD.

I'm not downplaying how amazing these capabilities are. Being able to generate high-quality code from natural language feels like magic. But all the parts beyond narrow application code are half of the thing I described:

* I'm saying you should be able to send a single off-the-cuff drunk text to an AI and later find a complete production-ready SaaS startup that fully aligns with a reasonable interpretation of your message.

* The other half of the whole thing is >=human-level execution. If the AI can't autonomously deliver work comparable to what an experienced CTO would (given the same requirements, an arbitrarily large hiring budget, and a stipulation to never contact you again until the work was done), it's not there yet.

Again, none of this is to dunk on agentic coding. My point is that I set an absurdly high bar because I want it to one day be met. Just as a $100 storage budget today is equivalent to $100m a few decades ago, I want to live to see a $100 engineering budget reach equivalency with last decade's $100m.

Re: Zebra-Llama – Towards efficient hybrid models

#59

Post by nl »

buu700 wrote: post #58
nl wrote: post #55

Earlier quoted context omitted.

Have you tried Loveable, Replit, V0 etc? Outside of purchasing the domain and native apps for you they cover a very significant amount of this. If you insist on Native Apps, it's possible Google Jules could do it. With Gemini 2.5 it wasn't strong enough but I think it has Gemini 3 now which can definitely do native apps just fine.

Thanks for the recommendations. Regarding your other comment, Flutter is what I've landed on as well for my next cross-platform app project, and I'm currently in the middle of developing a spec for a fairly complex agentic system that I'm going to try having Codex two-shot (basic project setup + file stubs + exhaustive tests -> manual checkpoint -> TDD the rest). I haven't tried Lovable, V0, or Jules, but I really li…

If you haven't tried these things since Sonnet 4.5 came out then it's time to give them another try.

Sonnet 4.5 and especially Codex 5.1 have completely changed the way I build software.

> The fact that tools like Replit also include their own hosting environments is definitely neat, but not really what I was getting at as far as deployment. What I had in mind was managing arbitrary cloud platforms, setting up an optimal architecture for your anticipated scale and usage patterns — whether that's a single Hetzner instance with SQLite or horizontally scaled app servers behind an API gateway with Kafka, Valkey, and Spanner or ScyllaDB — and doing all the DevOps to handle that along with things like CI/CD.

I think this is all possible now. But I don't think it'd work first time because there are so many environmental issues (service auth etc) that can go wrong. Maybe it'd be ok if you have it a root AWS account...

Re: Zebra-Llama – Towards efficient hybrid models

#60

Post by buu700 »

nl wrote: post #59
buu700 wrote: post #58

Earlier quoted context omitted.

Thanks for the recommendations. Regarding your other comment, Flutter is what I've landed on as well for my next cross-platform app project, and I'm currently in the middle of developing a spec for a fairly complex agentic system that I'm going to try having Codex two-shot (basic project setup + file stubs + exhaustive tests -> manual checkpoint -> TDD the rest). I haven't tried Lovable, V0, or Jules, but I really li…

If you haven't tried these things since Sonnet 4.5 came out then it's time to give them another try. Sonnet 4.5 and especially Codex 5.1 have completely changed the way I build software. > The fact that tools like Replit also include their own hosting environments is definitely neat, but not really what I was getting at as far as deployment. What I had in mind was managing arbitrary cloud platforms, setting up an opt…

Just in case it was unclear, I extensively use AI and agentic coding with current models on a daily basis. The only thing I haven't tried in a few months is specifically one-shotting a greenfield project.

I know computer-use agents exist, and theoretically have tooling and permission to do all the things a human sitting in front of a computer can. I just haven't heard of anyone successfully claiming to have had one do exactly what I described for a non-toy project in one shot with zero mistakes, or of any tool like Replit claiming to support such a capability.

I'd be very interested to know if my impression is out of date. As in, if I could send a single message to some AI service and say "Here's my credit card, banking info, and entity info/EIN; build me a production-ready Google Drive clone with religious branding and 10x higher pricing called God Drive with native Android/iOS/Linux/macOS/Windows apps, then deploy it to production on an optimal cloud architecture capable of scaling to a billion users at whatever domain name you like best and release the apps to all major app stores/repositories", then go to bed with high confidence that I'd be able to start creating God Drive docs/spreadsheets/presentations for work the following morning.

If that isn't the case, it isn't a criticism of the technology. The fact that we're even seriously discussing the scenario is incredible.

Post reply on HN