Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MAI-Code-1-Flash

microsoft.ai

Post reply on HN
Print view View original

211–220 of 297 posts

Re: MAI-Code-1-Flash

#211

Post by davecitron »

camelmel wrote: post #43

Huh, according to that model card this is a 137B total parameter model. Performance doesn't seem that good: - MAI-Code-1-Flash (137B-A5B) = 51% on SWE-bench pro - Qwen3.6-35B-A3B = 49.5% on SWE-bench pro ( https://huggingface.co/Qwen/Qwen3.6-35B-A3B ) They benchmark against Claude Haiku but Haiku is not good, it's worse than tiny open models you can run locally or via API at 10% the cost.

Dave Citron here, from the MAI team. Thanks for the feedback, we're getting the model card updated to call out 5B active parameters (137B total).

On benchmarks: in the same VS Code harness, MAI-Code-1-Flash scored 51.2% on SWE-bench Pro vs. Haiku's 35.2% which we see as a pretty big leap. But going forward, we'll include additional models in our benchmarks, including models like Qwen 3.6 and Gemma 4.

Re: MAI-Code-1-Flash

#212

Post by davecitron »

mdasen wrote: post #96
giancarlostoro wrote: post #57

Earlier quoted context omitted.

The take away is that this model is a smaller model that competes with Haiku, I would hope they come out with a "Sonnet" competing model, then Opus. I have been wondering why Microsoft is kind of "sleeping" on offering models they themselves have made on Copilot, maybe it was part of their deal with OpenAI? Not sure.

Yes, it's a "smaller" (137B) model that competes with Haiku, but it's basically the performance of Qwen3.6-35B-A3B which is 75% smaller and 98% smaller in terms of active parameters (since it's a mixture of experts model). Microsoft should be comparing its model to good smaller models, not Haiku 4.5. Qwen-3.6-27b is closer to Claude Opus 4.7 than it is to Haiku 4.5 in a lot of benchmarks - and it's way smaller than M…

[dead]

Re: MAI-Code-1-Flash

#213

Post by bitexploder »

ojr wrote: post #38
hmokiguess wrote: post #33

Does anyone actually uses these smaller models for coding? If so, how? I usually Opus everything. Is the play to plan/design/architect with a heavier model than delegate structured tasks to these smaller ones? Would appreciate to hear someone's opinion on having done and tested both paths.

I use Gemini 3 Flash, I've seen the Claude Code setups, bullish on Anthropic people are driving up tokens but I am able to produce outcomes with a fraction of the money.

3 Flash is likely rather underrated here. It continues to impress me on few-shot tasks.

Re: MAI-Code-1-Flash

#214

Post by epolanski »

camelmel wrote: post #43

Huh, according to that model card this is a 137B total parameter model. Performance doesn't seem that good: - MAI-Code-1-Flash (137B-A5B) = 51% on SWE-bench pro - Qwen3.6-35B-A3B = 49.5% on SWE-bench pro ( https://huggingface.co/Qwen/Qwen3.6-35B-A3B ) They benchmark against Claude Haiku but Haiku is not good, it's worse than tiny open models you can run locally or via API at 10% the cost.

So what other models use less than half of Haiku's tokens while providing higher success rate?

Re: MAI-Code-1-Flash

#215

Post by epolanski »

bel8 wrote: post #82

It's a start and I welcome competition but I don't think I ever used small cloud models like Haiku 4.5. They are cute but for serious coding they tend to waste your expensive time. And this certainly wont bring me back to GitHub Copilot which I cancelled yesterday. GitHub Copilot had competitive pricing until yesterday when they changed from per-request to one of the most expensive per-token quotas. Seriously, take a…

> They are cute but for serious coding they tend to waste your expensive time.

90% of corporate job tasks are trivial enough that Haiku can handle them.

Just this morning I have been implementing a reprint functionality in our warehouse management system, which needed to print again carrier labels and delivery notes for a specific order.

It essentially had to do the same workflow of print, but instead of generating and uploading the pdfs, it only had to fetch and print them.

Took Opus 4.8 high 24m1 seconds and 87k tokens. Took Haiku 6m30 seconds and half the tokens.

So not really sure what do you mean by "wasting your expensive time" here. I think you really don't experiment with these tools and assume higher effort, bigger model => time saved, but that's true only when tasks are much bigger and complex enough that a smaller/less precise model would fail or land work of much lower quality.

Re: MAI-Code-1-Flash

#216

Post by akie »

epolanski wrote: post #214
camelmel wrote: post #43

Huh, according to that model card this is a 137B total parameter model. Performance doesn't seem that good: - MAI-Code-1-Flash (137B-A5B) = 51% on SWE-bench pro - Qwen3.6-35B-A3B = 49.5% on SWE-bench pro ( https://huggingface.co/Qwen/Qwen3.6-35B-A3B ) They benchmark against Claude Haiku but Haiku is not good, it's worse than tiny open models you can run locally or via API at 10% the cost.

So what other models use less than half of Haiku's tokens while providing higher success rate?

Why is Haiku the benchmark though, with code generation don't we primarily care about the quality of the code - not the speed or efficiency at which it's generated?

Re: MAI-Code-1-Flash

#217

Post by epolanski »

akie wrote: post #216
epolanski wrote: post #214

Earlier quoted context omitted.

So what other models use less than half of Haiku's tokens while providing higher success rate?

Why is Haiku the benchmark though, with code generation don't we primarily care about the quality of the code - not the speed or efficiency at which it's generated?

I'm not sure I follow, but I'll give you a very fresh example.

I was implementing a re-print functionality in my warehouse management system.

It took Opus 4.8 high 24m1s and 87k tokens. Took Haiku 6m30s and 41k tokens.

After that time I had to provide (minor) adjustments to both. But Haiku allowed me to iterate faster. Code quality for that somewhat trivial use case was similar.

Actually, I would even say that Opus provided a sub par solution: instead of fixing an issue where carrier label pdf wasn't saved as the state machine progressed to the latest step, it went through a much complex solution of re-generating those by scratch. Which is also wrong, as it was de-facto booking the carriers twice for the same order.

Haiku simply added another field on the terminal state that carried the already generated urls.

I don't think it's a good idea to default to highest effort/bigger model without taking into account the time it takes and the task complexity.

Imho we should experiment rather than assume that what the rest of the community does to be the best practice.

Re: MAI-Code-1-Flash

#218

Post by zoobab »

"It is built end-to-end by Microsoft using clean and appropriately licensed data."

Well still no list nor publication of the training data.

Re: MAI-Code-1-Flash

#220

Post by NitpickLawyer »

akie wrote: post #216
epolanski wrote: post #214

Earlier quoted context omitted.

So what other models use less than half of Haiku's tokens while providing higher success rate?

Why is Haiku the benchmark though, with code generation don't we primarily care about the quality of the code - not the speed or efficiency at which it's generated?

You would be surprised how much code haiku writes behind the scenes. With the whole 'plan w/ opus, spawn subagents w/ haiku' that cc does. And you'd be surprised how useful the small models can be under some guidance / hand holding. You can daily-drive gpt5-mini and still find it useful. They're not as good as the big ones, obviously, and can't handle a project start-to-finish on their own, but given a well-scoped task, they'll do it just fine.
Post reply on HN