Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K3 Is Competitive with Fable; Kimi K3 and Fable Is SoTA

fireworks.ai

Post reply on HN
Print view View original

451–460 of 491 posts

Re: Kimi K3 Is Competitive with Fable; Kimi K3 and Fable Is SoTA

#451

Post by maxignol »

makingstuffs wrote: post #402
senko wrote: post #380

Earlier quoted context omitted.

For the web app task I mentioned: * Kimi K3: 9532k input (9172k cached), 114k output - cost $5.5 * Qwen 3.8 Max: 18020k input (17836k cached), 114k output - cost $6.3 * Fable: ~14m input (all cached??), 196k output - cost $30 Correction on my earlier post, Kimi was through Pi, not Kimi Code. For Qwen I used Qwen Code and for Fable I used Claude Code. Not sure wtf is going on with the Fable stats (a lot tokens, virtua…

Would a fairer test not be to use the same harness for all three? I’d suspect the harness to massively affect token use and optimisation

> I’d suspect the harness to massively affect token use and optimisation

Yes they do according to databricks -> https://www.databricks.com/blog/benchmarking-coding-agents-d...

That's why I find comparing models on benchmarks only gives the tendency. We should be comparing model x harness to have accurate metrics.

Re: Kimi K3 Is Competitive with Fable; Kimi K3 and Fable Is SoTA

#452

Post by lumost »

maxignol wrote: post #446
lumost wrote: post #409

Earlier quoted context omitted.

Honestly my anecdotal experience is that fable is benchmaxxed. I have not observed useful gains for Claude since 4.6, with each model iteration making progressively poorer decisions in pursuit of its goal. The 5.5/5.6 series has performed quite well however. My guess is that my use cases stop aligning to swebench pro around 50% accuracy, and more closely align with DeepSWE.

> my use cases stop aligning to swebench pro around 50% accuracy, and more closely align with DeepSWE. What do you mean by that ? If the model is higher than 50% on swebench pro then it tends to drift from what you like it to do, like DeepSWE benchmarks ?

yes, above 50% I don't observe a significant improvement, the models scoring above 65% like opus4.8 seem worse than gpt5.5/opus4.6. DeepSWE appears to align more closely to my personal experience.

Re: Kimi K3 Is Competitive with Fable; Kimi K3 and Fable Is SoTA

#453

Post by makingstuffs »

JaRail wrote: post #411
makingstuffs wrote: post #402

Earlier quoted context omitted.

Would a fairer test not be to use the same harness for all three? I’d suspect the harness to massively affect token use and optimisation

Yes and no. It would make the test fair but it would also mean losing any optimizations the vendors have made specifically for their model. They're all trained differently so they should be used differently to get maximum benefit. It makes sense to use each vendor's harness to get the most of their models, and compare the results that way. Additionally, it's not easy to use all the models in all the harnesses. Anthro…

I don’t know that the methodology of the experiment is testing what it intends to test. With the current method we’re essentially testing each AI lab’s ability to efficiently extract data from their version of a transformer model.

To objectively test all models the harness would need to be the same and ideally independent. Failing that, all three models should be tested in all three harnesses and the output verified on a model x harness level and on an overall aggregated model x all harnesses level.

Re: Kimi K3 Is Competitive with Fable; Kimi K3 and Fable Is SoTA

#454

Post by makingstuffs »

maxignol wrote: post #451
makingstuffs wrote: post #402

Earlier quoted context omitted.

Would a fairer test not be to use the same harness for all three? I’d suspect the harness to massively affect token use and optimisation

> I’d suspect the harness to massively affect token use and optimisation Yes they do according to databricks -> https://www.databricks.com/blog/benchmarking-coding-agents-d... That's why I find comparing models on benchmarks only gives the tendency. We should be comparing model x harness to have accurate metrics.

Agreed. An experiment is only as accurate as the methodology is sound.

Testing each lab’s model in its own harness just tells us how well the lab has performed. For the model’s performance we need a control and the only way to get the control is to either test all models in all harnesses or all models in the same harness

Re: Kimi K3 Is Competitive with Fable; Kimi K3 and Fable Is SoTA

#455

Post by rescbr »

beng-nl wrote: post #359
fractalf wrote: post #338

Earlier quoted context omitted.

You tell jokes to your model? :D

Not op, but I do sometimes indulge in such anthropomorphic conversation. Confiding in it that a certain (bad) result in the research project we’re working on ‘feels bad’ and reading its supportive reply makes me feel less alone in failure. In another instance, ChatGPT didn’t think a particular test would prove to be statistically significant, so I ‘bet’ with it it would (after collecting an agreed on number of sample…

I also make nerdy jokes/puns, and I had similar experiences with beneficial model steering that such puns create.

It's interesting how more loose/informal prompting achieves good results, there is some cultural understanding in the models from training. Once I asked the clanker to remove the gambiarras and puxadinhos that it wrote as part of an experiment, and it promptly fixed those.

Re: Kimi K3 Is Competitive with Fable; Kimi K3 and Fable Is SoTA

#456

Post by bmitc »

dbspin wrote: post #360
bmitc wrote: post #325

Earlier quoted context omitted.

Why is token efficiency a concern with free models?

They're not free to run, Kimi K3 needs to be run on the cloud, and the quantised versions aren't as capable. Unless you happen to have 3 - 5 TB of VRAM and an 8-node cluster of 8× NVIDIA H100s to run the full fat version. Plus the weights are not yet available to download in any case.

Qwen was mentioned in the comment I replied to and can run locally.

Re: Kimi K3 Is Competitive with Fable; Kimi K3 and Fable Is SoTA

#457

Post by w4yai »

llelouch wrote: post #283
jeffreyrogers wrote: post #272

Earlier quoted context omitted.

Fable works very well for me on a moderately large codebase. I have had to correct it a few times or point it on the right track, but given how much faster it is at programming than I am that's a very minor issue (and most of these errors are because I underspecified what I wanted in the prompt, I can only think of two cases where it was genuinely wrong... that's a lot better than me in my professional career). Code…

Fable is the clearly best when you have to do real coding.

Define "real coding" ?

For low-level x86 assembly coding, Fable is nowhere near to be as good as Kimi K3.

Re: Kimi K3 Is Competitive with Fable; Kimi K3 and Fable Is SoTA

#458

Post by Der_Einzige »

dev_hugepages wrote: post #328
Der_Einzige wrote: post #262

Earlier quoted context omitted.

You got baited by bad sampling settings. It's exactly the opposite. Go turn on min_p once it's available post July 27th and most of the problems you describe will go away.

What parameter would you advise for min_p?

as temperature approaches infinity, min_p must approach 1 to stay coherent.

Assuming your temp is below 2, min_p of 0.1 is fine (and disable top_p and top_k). You can try 0.05 for more diversity.

Remember that subsequent methods are better, min_p is a mid-tier sampler that just happens to be the best implemented in most inference providers right now.

Also I'm the author of the "conspiracy against high temperature sampling" thing that selfhoster posted in the comments, so you can ask any questions about that piece you want.

Re: Kimi K3 Is Competitive with Fable; Kimi K3 and Fable Is SoTA

#459

Post by simjnd »

paulddraper wrote: post #437
simjnd wrote: post #431

Earlier quoted context omitted.

> the Chinese models really are slow and token-inefficient This totally depends on the model. Deepseek V4 is very fast and efficient.

Not close to frontier. But yes, it’s cheap.

The comment I replied to didn't mention anything about being close to frontier, just a blanket statement about Chinese Labs models being slow and inefficient.

People talk about frontier as if it's the only innovation worth pursuing. Deepseek V4 is far from fontier, but it's architecture is super innovative and efficient and what it achieves at that size (especially V4 Flash) is incredible.

Re: Kimi K3 Is Competitive with Fable; Kimi K3 and Fable Is SoTA

#460

Post by paulddraper »

simjnd wrote: post #459
paulddraper wrote: post #437

Earlier quoted context omitted.

Not close to frontier. But yes, it’s cheap.

The comment I replied to didn't mention anything about being close to frontier, just a blanket statement about Chinese Labs models being slow and inefficient. People talk about frontier as if it's the only innovation worth pursuing. Deepseek V4 is far from fontier, but it's architecture is super innovative and efficient and what it achieves at that size (especially V4 Flash) is incredible.

> Having tested K3, Qwen 3.8 max preview, Fable and Sol for the past few days, the Chinese models really are slow and token-inefficient.

Those are all frontier-competitive models.

Post reply on HN