Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K3, and what we can still learn from the pelican benchmark

simonwillison.net

Post reply on HN
Print view View original

171–180 of 245 posts

Re: Kimi K3, and what we can still learn from the pelican benchmark

#171

Post by simonw »

not_a_bot_4sho wrote: post #164
simonw wrote: post #77

Earlier quoted context omitted.

I've done versions in the past where I ran 3 and picked the best one. At some point I'd like to automate that with an LLM-as-a-judge (from the same model family) picking the "best" one to move forth in the competition. I built a whole ELO scoring mechanism a while back, described here: https://simonwillison.net/2025/Jun/6/six-months-in-llms/#ai-... I probably should spend some time on this now, even though the benchm…

If you're not doing *at least* say 100 iterations (thousands are preferred!!), you do not have enough data to draw any stable conclusions. Interestingly enough, using an LLM-as-judge is a great way to approach things like this at scale but you do need to invest in some Cohen's Kappa or Fleiss' Kappa understanding which means putting a human in the driver seat to evaluate the effectiveness of your non-human judge. Abs…

I'm not sure there's any level of iterations that could result in a credible decision that model A clearly draws a better pelican riding a bicycle than model B.

What does "better" even mean there?

Re: Kimi K3, and what we can still learn from the pelican benchmark

#172

Post by not_a_bot_4sho »

simonw wrote: post #171
not_a_bot_4sho wrote: post #164

Earlier quoted context omitted.

If you're not doing *at least* say 100 iterations (thousands are preferred!!), you do not have enough data to draw any stable conclusions. Interestingly enough, using an LLM-as-judge is a great way to approach things like this at scale but you do need to invest in some Cohen's Kappa or Fleiss' Kappa understanding which means putting a human in the driver seat to evaluate the effectiveness of your non-human judge. Abs…

I'm not sure there's any level of iterations that could result in a credible decision that model A clearly draws a better pelican riding a bicycle than model B. What does "better" even mean there?

(I came to delete but was too late. So edits are in.)

Wow, that's a stark take. I suppose I'm biased towards a scientific viewpoint. All the best.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#173

Post by xigoi »

oceanplexian wrote: post #34
OsrsNeedsf2P wrote: post #13

It's incredible Simon still believes pelicans on bikes aren't part of the training set, despite hundreds of them on blogs, forums, and Github. Stuff we put in our company blog shows up known by LLMs 6 months later, and we have 1000x less traffic than Simon's own website

Imagine if we applied this train of logic to humans. "That artist saw a pelican at the beach once!" [cue the outrage] "He's not a real artist, he's a cheater and produces nothing original!"

More like “This artist won the drawing competition because someone told her the theme in advance and the specifically practiced drawing pelicans for hundreds of hours.”

Re: Kimi K3, and what we can still learn from the pelican benchmark

#174

Post by dannyw »

tibbar wrote: post #45

I wonder how the Chinese labs are training a 3 trillion parameter model on what has to be vastly smaller compute resources. If the U.S. compute advantage is persistent, it's hard to imagine that Chinese labs will be able to keep pace forever, as a matter of physics, but... so far they seem to be doing just fine.

Gamers Nexus has a good video where his team travelled to China and did some actual investigative journalism: https://www.youtube.com/watch?v=1H3xQaf7BFI

Firstly, the export-restricted GB202s (e.g. 5090, RTX 6000 Pro Blackwell) are fabled in TSMC, and then packaged/made in... China before they supposedly have to be sold out (by US law; but not by Chinese law). You can immediately see the problem there.

Secondly, despite the supposed 'crackdowns' and et al, NVIDIA and their channel partners pretty much will sell to anyone in countries like Singapore without any questions.

Third, there's human "smugglers" who just physically carry em on trips, and Chinese customs is obviously not going to care about the US's laws on Chinese soil.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#175

Post by dannyw »

lambda wrote: post #79
Lerc wrote: post #30

Do any of the vision models render the SVG and look at the result. Perhaps more importantly can they do that during reinforcement training. Learning how to critically analyse the appearance of what it generates would be quite useful. Manually feeding images back to models has been hilariously bad in the past which suggests that relating something it sees to something it wrote is not an ability it is very good at.

I've tried doing a loop of rending the SVG and then tweaking based on that, with local models (so, not nearly as strong). It wasn't very successful; it would mostly report that the image looked great and didn't need any tweaks. Maybe I should try it again, there have been some newer models since I first tried it. And yeah, maybe worth trying with bigger models. But I have found that models aren't necessarily the best…

I've found much better luck giving it an audit check-list, including some steers like: are there any visual glitches or SVG bugs, are the colours consistent, etc.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#176

Post by dannyw »

Creamsicle47 wrote: post #133
esafak wrote: post #128

Earlier quoted context omitted.

You have to look at the size of each expert; Kimi's has about 50G parameters while GLM's has 40G. The number of the experts tells you about the diversity of its skills.

> You have to look at the size of each expert Yes, this part is accurate. Expert density determines how much raw compute each hidden state gets. > The number of the experts tells you about the diversity of its skills. Most people misunderstand this part. Counter-intuitively experts don't develop diverse skills, they instead balance compute during the forward pass, allowing models to increase their parameter count wit…

Yeah, "experts" is a ML/research word for this (MoE was first published in 1991; and has been around for a long time, it even predates deep learning). it's not the everyday/colloquial meaning of 'expert'.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#177

Post by dannyw »

criddell wrote: post #62
nothercastle wrote: post #51

It’s not bad kind of expensive for 25c but if the prompt is rendered cost is much better.

I wonder what the non-subsidized cost is. Add in the electricity and water too. We may be boiling the oceans but at least we are finally getting some good SVGs of pelicans on bicycles.

We're looking at a MoE with 50B active params, each inference pass only requires the compute of a 50B dense model.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#178

Post by dannyw »

mrcwinn wrote: post #23

K3 is as expensive as Sonnet, not great at writing English, is handing IP back to the Chinese, and once open source will be difficult to run at scale without the compute that OpenAI and Anthropic have largely grabbed. Sorry, how again is this the end of the frontier labs?

Well, with the actions of the US government, for every business that does not exclusively operate in the US, they have now added _supplier risk_ to US companies.

Even as a paying customer, even as an enterprise, your access to US models may be turned off at any time for arbitrary reasons, including someone mis-understanding "Please fix this [open source] code" (which contained security vulnerabilities that were fixed) as a jailbreak.

Post reply on HN