Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K3, and what we can still learn from the pelican benchmark

simonwillison.net

Post reply on HN
Print view View original

161–170 of 245 posts

Re: Kimi K3, and what we can still learn from the pelican benchmark

#161

Post by jambutters »

somelamer567 wrote: post #135

I'm wondering what the grift here is. Usually, the pattern is that we see a tsunami of planted "China number one" stories boosted by hordes of Chinese "internet commentators", and then the world trembles for a few days until the scam mechanics are revealed. My would be either: crippling limitations on the model, vast, unfair, and/or illegal subsidies by the CCP regime as a mercantilist attack on Western capabilities…

I think this is one of the few cases where there isn't a grift. It's open source, open research, there's not much to hide? Honestly official statements are pretty tame, it's the people who spin them for media headline clicks that are warping reality

Re: Kimi K3, and what we can still learn from the pelican benchmark

#162

Post by swyx »

btown wrote: post #140

> The biggest limitation of the pelican is that it doesn’t touch at all on the thing that matters most for today’s model: agentic tool calling and the ability to operate tools reliably as conversations grow in length. In all seriousness, I propose SWE-bench-adversarial-pelican-gen: it's like SWE-bench, but the harness gets interrupted every 5 turns/tool-calls and is asked to produce an SVG of an arbitrary animal befo…

this is probably about $5 bucks in codex . worth introspecting why nobody seems excited to run it

Re: Kimi K3, and what we can still learn from the pelican benchmark

#163

Post by sm0ss117 »

The disconnection between pelican quality and overall model quality is interesting. I initially assumed that since pre-training is when a model gets its general skill that it happened around when RL started to really differentiate models. That is higher quality pre-trains result in higher quality pelicans, but RL is unlikely to touch pelican quality. However the fact that GLM 5.2 beats GPT 5.6 and Claude Fable puts a damper on that idea.

My only guess is that GLM 5.2 was specifically RLed for SVG generation and that resulted in superior performance.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#164

Post by not_a_bot_4sho »

simonw wrote: post #77
yashchimata wrote: post #76

One thing i keep thinking: you only run the pelican once per model. Run the same model a few times and you get some different pelicans, so some of "this one is better" might just be which run you picked for it. Would love to see 8 runs per model side by side. I bet for two close models, the gap between runs is about as big as the gap between the models.

I've done versions in the past where I ran 3 and picked the best one. At some point I'd like to automate that with an LLM-as-a-judge (from the same model family) picking the "best" one to move forth in the competition. I built a whole ELO scoring mechanism a while back, described here: https://simonwillison.net/2025/Jun/6/six-months-in-llms/#ai-... I probably should spend some time on this now, even though the benchm…

If you're not doing *at least* say 100 iterations (thousands are preferred!!), you do not have enough data to draw any stable conclusions.

Interestingly enough, using an LLM-as-judge is a great way to approach things like this at scale but you do need to invest in some Cohen's Kappa or Fleiss' Kappa understanding which means putting a human in the driver seat to evaluate the effectiveness of your non-human judge. Absent of that, it's just another case of human-centipede but with LLMs.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#165

Post by jhalloran »

OsrsNeedsf2P wrote: post #13

It's incredible Simon still believes pelicans on bikes aren't part of the training set, despite hundreds of them on blogs, forums, and Github. Stuff we put in our company blog shows up known by LLMs 6 months later, and we have 1000x less traffic than Simon's own website

This reminded me about the news cycle last year that we were running out of training data (and how silly that was)

Re: Kimi K3, and what we can still learn from the pelican benchmark

#166

Post by [deleted] »

OsrsNeedsf2P wrote: post #13

It's incredible Simon still believes pelicans on bikes aren't part of the training set, despite hundreds of them on blogs, forums, and Github. Stuff we put in our company blog shows up known by LLMs 6 months later, and we have 1000x less traffic than Simon's own website

[deleted]

Re: Kimi K3, and what we can still learn from the pelican benchmark

#167

Post by [deleted] »

OsrsNeedsf2P wrote: post #13

It's incredible Simon still believes pelicans on bikes aren't part of the training set, despite hundreds of them on blogs, forums, and Github. Stuff we put in our company blog shows up known by LLMs 6 months later, and we have 1000x less traffic than Simon's own website

[deleted]

Re: Kimi K3, and what we can still learn from the pelican benchmark

#168

Post by Jimmc414 »

OsrsNeedsf2P wrote: post #13

It's incredible Simon still believes pelicans on bikes aren't part of the training set, despite hundreds of them on blogs, forums, and Github. Stuff we put in our company blog shows up known by LLMs 6 months later, and we have 1000x less traffic than Simon's own website

[dead]

Re: Kimi K3, and what we can still learn from the pelican benchmark

#169

Post by [deleted] »

simonw wrote: post #77
yashchimata wrote: post #76

One thing i keep thinking: you only run the pelican once per model. Run the same model a few times and you get some different pelicans, so some of "this one is better" might just be which run you picked for it. Would love to see 8 runs per model side by side. I bet for two close models, the gap between runs is about as big as the gap between the models.

I've done versions in the past where I ran 3 and picked the best one. At some point I'd like to automate that with an LLM-as-a-judge (from the same model family) picking the "best" one to move forth in the competition. I built a whole ELO scoring mechanism a while back, described here: https://simonwillison.net/2025/Jun/6/six-months-in-llms/#ai-... I probably should spend some time on this now, even though the benchm…

[deleted]

Re: Kimi K3, and what we can still learn from the pelican benchmark

#170

Post by crazylogger »

kllrnohj wrote: post #131
tibbar wrote: post #45

I wonder how the Chinese labs are training a 3 trillion parameter model on what has to be vastly smaller compute resources. If the U.S. compute advantage is persistent, it's hard to imagine that Chinese labs will be able to keep pace forever, as a matter of physics, but... so far they seem to be doing just fine.

Or they just don't actually have any compute access restrictions of significance? Chinese companies can just go use those GPUs in neighboring countries that aren't export-restricted, like Malaysia. Like ByteDance openly did: https://www.tomshardware.com/pc-components/gpus/chinas-byted... and Tencent is rumored to have done via Japan: https://wccftech.com/china-tencent-gains-access-to-nvidia-bl... And that's not even…

Even ignoring chip export ban, Chinese companies have way less funding than American counterparts, maybe 1 or 2 orders of magnitude less depending on which company you look at. Deepseek’s recent big funding round being “only” a couple billion $ at $50B valuation, for example. Bytedance and Tencent are tech giants for sure, nonetheless they’re not Google kind of giant.
Post reply on HN