Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K3, and what we can still learn from the pelican benchmark

simonwillison.net

Post reply on HN
Print view View original

131–140 of 245 posts

Re: Kimi K3, and what we can still learn from the pelican benchmark

#131

Post by kllrnohj »

tibbar wrote: post #45

I wonder how the Chinese labs are training a 3 trillion parameter model on what has to be vastly smaller compute resources. If the U.S. compute advantage is persistent, it's hard to imagine that Chinese labs will be able to keep pace forever, as a matter of physics, but... so far they seem to be doing just fine.

Or they just don't actually have any compute access restrictions of significance? Chinese companies can just go use those GPUs in neighboring countries that aren't export-restricted, like Malaysia. Like ByteDance openly did: https://www.tomshardware.com/pc-components/gpus/chinas-byted...

and Tencent is rumored to have done via Japan: https://wccftech.com/china-tencent-gains-access-to-nvidia-bl...

And that's not even considering just smuggling the GPUs in by eg buying them in Singapore.

AI-specific chips also seem to be on the easier side to design & create relative to high performance CPUs & GPUs, so there's no particular reason to expect Chinese domestic designs to continuously lag behind. They have access to the same fabs, after all

Re: Kimi K3, and what we can still learn from the pelican benchmark

#133

Post by Creamsicle47 »

esafak wrote: post #128
andai wrote: post #64

3T is impressive, but parameter count seems to be less important than I thought. GLM is half the size of DeepSeek but costs four times as much, and beats it on every benchmark. I'm not an expert on this stuff but it seems to be the attention mechanism. DeepSeek were bragging about how cheap they made it. But if you cut costs on attention you get worse results with way more parameters. If I had to guess it seems to be…

You have to look at the size of each expert; Kimi's has about 50G parameters while GLM's has 40G. The number of the experts tells you about the diversity of its skills.

> You have to look at the size of each expert

Yes, this part is accurate. Expert density determines how much raw compute each hidden state gets.

> The number of the experts tells you about the diversity of its skills.

Most people misunderstand this part. Counter-intuitively experts don't develop diverse skills, they instead balance compute during the forward pass, allowing models to increase their parameter count without the MLP layers exploding in memory + compute requirements.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#134

Post by pclmulqdq »

ahtihn wrote: post #94
pterhx wrote: post #90

Earlier quoted context omitted.

In photography (and probably art in general), there's a composition "rule" to frame moving subjects from left to right. So the direction may not be that interesting!

I wonder if that changes in countries where the main language is written right to left?

It is. All over the Arab world, imagery in ads is “backwards” and I believe several companies will flip their ads horizontally, and UI localization involves flipping graphics.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#135

Post by somelamer567 »

I'm wondering what the grift here is.

Usually, the pattern is that we see a tsunami of planted "China number one" stories boosted by hordes of Chinese "internet commentators", and then the world trembles for a few days until the scam mechanics are revealed.

My would be either: crippling limitations on the model, vast, unfair, and/or illegal subsidies by the CCP regime as a mercantilist attack on Western capabilities (as we've already seen in iron smelting and clean energy), sanctions-busting, gamed benchmarks, outright theft -- or a combination of the above.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#136

Post by conception »

OtherShrezzing wrote: post #75
simonw wrote: post #66

Earlier quoted context omitted.

The pelicans are still all rubbish. If they make it into the training set it doesn't help the models produce better pelicans, if anything it will make them perform worse!

Respectfully, the pelicans used to be an unrecognisable mess and now they’re unquestionably pelicans on bicycles, rendered poorly, from every model. In the same timescale, model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts. Moreover, they have a uniform style, even though your prompt doesn’t ask for one. There's no model going rogue and pro…

If you’ve been keeping track of all of the pelicans, there is actually stylistic differences - sometimes pretty big differences as far as watercolors go. It’s an SVG so I’m not sure what you’re looking for there. Most look the same because the prompt is to make a pelican on a bicycle as an SVG. It’s not some giant image prompt.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#137

Post by sroussey »

simonw wrote: post #120
danielrmay wrote: post #117

Earlier quoted context omitted.

What does good look like?

The dedicated text-to-image models all produce good illustrations of pelicans riding bicycles. Here's one I got from OpenAI's gpt-image-2 just the other day: https://simonwillison.net/2026/Jul/14/pedalican/

Try prompts that convert the gif to svg? Maybe the models should do that internally… start with their image model first and then make an svg.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#138

Post by sneurlax »

somelamer567 wrote: post #135

I'm wondering what the grift here is. Usually, the pattern is that we see a tsunami of planted "China number one" stories boosted by hordes of Chinese "internet commentators", and then the world trembles for a few days until the scam mechanics are revealed. My would be either: crippling limitations on the model, vast, unfair, and/or illegal subsidies by the CCP regime as a mercantilist attack on Western capabilities…

Invest in energy, manufacturing, and education (ie. your own people) for 75 years and people will look for a trick card up your sleeve and accuse you of cheating when your 7th of the world population has a 7th of the world's genius

Re: Kimi K3, and what we can still learn from the pelican benchmark

#139

Post by dolebirchwood »

dgellow wrote: post #72
whywhywhywhy wrote: post #40

Don't see why we have to have this spammed every model release when Fable class models perform the same as Opus on basic tasks like these.

What spam? It’s one article. You can skip it

One article... every time. And the only reason it gets any traction is because of who the author is -- not because of anything substantively useful. Do you think this whole "pelican on a bicycle" would have blown up if, say, you were the first?

Re: Kimi K3, and what we can still learn from the pelican benchmark

#140

Post by btown »

> The biggest limitation of the pelican is that it doesn’t touch at all on the thing that matters most for today’s model: agentic tool calling and the ability to operate tools reliably as conversations grow in length.

In all seriousness, I propose SWE-bench-adversarial-pelican-gen: it's like SWE-bench, but the harness gets interrupted every 5 turns/tool-calls and is asked to produce an SVG of an arbitrary animal before being told to continue, and every few tool call outputs add comment lines that refer to SVGs of pelicans (and, perhaps, how a møøse bit my sister once). And, at the end, once it's 800k tokens deep into context, it's asked to produce an SVG of a pelican and is evaluated against both the pelican and the completion and efficiency of the task.

You're only as good as your ability to solve problems in the midst of an SVG pelican attack.

Post reply on HN