Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K3, and what we can still learn from the pelican benchmark

simonwillison.net

Post reply on HN
Print view View original

121–130 of 245 posts

Re: Kimi K3, and what we can still learn from the pelican benchmark

#121

Post by jefftk »

OtherShrezzing wrote: post #75
simonw wrote: post #66

Earlier quoted context omitted.

The pelicans are still all rubbish. If they make it into the training set it doesn't help the models produce better pelicans, if anything it will make them perform worse!

Respectfully, the pelicans used to be an unrecognisable mess and now they’re unquestionably pelicans on bicycles, rendered poorly, from every model. In the same timescale, model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts. Moreover, they have a uniform style, even though your prompt doesn’t ask for one. There's no model going rogue and pro…

> model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts

That doesn't seem right. I use these models as research assistants when writing lots of random blog posts (including in economically ~useless areas like the history of contra dance) and Fable 5 is a serious improvement (when I don't get downgraded!) over Opus 4.6-4.8 which was a serious improvement over Opus 4.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#123

Post by valleyer »

simonw wrote: post #103
copperx wrote: post #92

Earlier quoted context omitted.

Is it culture dependent? Is it because in English we read left to right?

There was a glorious moment when I thought that the Chinese models were more likely to produce right-to-left cycling pelicans, but sadly that trend didn't seem to hold up.

Arabic or Hebrew would be better tests for that.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#124

Post by choilive »

Anyone have any idea what the architecture/vendors they are using for inference/compute?

Getting the compute to run inference for multi-trillion parameter models at any sort of scale and performance is daunting. There are a handful of vendors that have systems that can do this (~ Nvidia NVl-72 class) that pretty much only the frontier labs and hyperscalers effectively have access to.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#125

Post by danielrmay »

simonw wrote: post #120
danielrmay wrote: post #117

Earlier quoted context omitted.

What does good look like?

The dedicated text-to-image models all produce good illustrations of pelicans riding bicycles. Here's one I got from OpenAI's gpt-image-2 just the other day: https://simonwillison.net/2026/Jul/14/pedalican/

Have you tried asking for an SVG with the same level of detail you specified in that prompt, though? https://github.com/simonw/pedalican/blob/main/run/prompts/ro...

I'd be interested to see what comes out, but it also highlights an curious prompt-control-comparison question

Re: Kimi K3, and what we can still learn from the pelican benchmark

#126

Post by pietz »

andai wrote: post #64

3T is impressive, but parameter count seems to be less important than I thought. GLM is half the size of DeepSeek but costs four times as much, and beats it on every benchmark. I'm not an expert on this stuff but it seems to be the attention mechanism. DeepSeek were bragging about how cheap they made it. But if you cut costs on attention you get worse results with way more parameters. If I had to guess it seems to be…

It's almost like they priced models based on their performance or something...

Re: Kimi K3, and what we can still learn from the pelican benchmark

#127

Post by Lerc »

cherioo wrote: post #59
Lerc wrote: post #30

Do any of the vision models render the SVG and look at the result. Perhaps more importantly can they do that during reinforcement training. Learning how to critically analyse the appearance of what it generates would be quite useful. Manually feeding images back to models has been hilariously bad in the past which suggests that relating something it sees to something it wrote is not an ability it is very good at.

I imagine all vision models have to do this, this being html rendering, to be able to do well in web design.

> to be able to do well in web design.

That's kind-of why I don't think they're doing that. Anything beyond something that works with a simple design templates looks, well, like they tried to do too much with a simple design template.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#128

Post by esafak »

andai wrote: post #64

3T is impressive, but parameter count seems to be less important than I thought. GLM is half the size of DeepSeek but costs four times as much, and beats it on every benchmark. I'm not an expert on this stuff but it seems to be the attention mechanism. DeepSeek were bragging about how cheap they made it. But if you cut costs on attention you get worse results with way more parameters. If I had to guess it seems to be…

You have to look at the size of each expert; Kimi's has about 50G parameters while GLM's has 40G. The number of the experts tells you about the diversity of its skills.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#129

Post by kristofferR »

tibbar wrote: post #45

I wonder how the Chinese labs are training a 3 trillion parameter model on what has to be vastly smaller compute resources. If the U.S. compute advantage is persistent, it's hard to imagine that Chinese labs will be able to keep pace forever, as a matter of physics, but... so far they seem to be doing just fine.

The Chinese just saved the world economy by draining their absurdly enormous oil storage reserves nobody knew they had, wouldn't surprise me if they had lots of hidden compute too.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#130

Post by elashri »

filoleg wrote: post #118
ahtihn wrote: post #94

Earlier quoted context omitted.

I wonder if that changes in countries where the main language is written right to left?

That was my first thought too, I wonder if it works the same in countries speaking arabic (as that's the first one i could think of that's a language with truly no-buts right to left writing).

Arabic native speaker here.

Yes, people will usually post or draw a bicycle right to left which is going to ve opposite of what normally is drawn. I tried the prompt in arabic for many models and I don't recall any adjusting it based on that difference at least culturally speaking.

Post reply on HN