Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K3, and what we can still learn from the pelican benchmark

simonwillison.net

Post reply on HN
Print view View original

181–190 of 245 posts

Re: Kimi K3, and what we can still learn from the pelican benchmark

#181

Post by epolanski »

crazylogger wrote: post #170
kllrnohj wrote: post #131

Earlier quoted context omitted.

Or they just don't actually have any compute access restrictions of significance? Chinese companies can just go use those GPUs in neighboring countries that aren't export-restricted, like Malaysia. Like ByteDance openly did: https://www.tomshardware.com/pc-components/gpus/chinas-byted... and Tencent is rumored to have done via Japan: https://wccftech.com/china-tencent-gains-access-to-nvidia-bl... And that's not even…

Even ignoring chip export ban, Chinese companies have way less funding than American counterparts, maybe 1 or 2 orders of magnitude less depending on which company you look at. Deepseek’s recent big funding round being “only” a couple billion $ at $50B valuation, for example. Bytedance and Tencent are tech giants for sure, nonetheless they’re not Google kind of giant.

Chinese companies likely aren't paying millions/year for their researchers but a tenth of it.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#183

Post by 0xbadcafebee »

tibbar wrote: post #45

I wonder how the Chinese labs are training a 3 trillion parameter model on what has to be vastly smaller compute resources. If the U.S. compute advantage is persistent, it's hard to imagine that Chinese labs will be able to keep pace forever, as a matter of physics, but... so far they seem to be doing just fine.

Huawei Ascend chips were used to train DeepSeek v4 over 4 months ago, and they shared their kernel with the other Chinese labs. China also has their own DDR5 fabs.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#185

Post by flir »

simonw wrote: post #66
OsrsNeedsf2P wrote: post #13

It's incredible Simon still believes pelicans on bikes aren't part of the training set, despite hundreds of them on blogs, forums, and Github. Stuff we put in our company blog shows up known by LLMs 6 months later, and we have 1000x less traffic than Simon's own website

The pelicans are still all rubbish. If they make it into the training set it doesn't help the models produce better pelicans, if anything it will make them perform worse!

It's just a gut feeling, but I think you're running a (very slow) distributed hill-climbing algorithm. LLM1 generates an SVG. You post it online, with commentary on what is good/bad about it. LLM2 consumes the SVG alongside your commentary, and produces a slightly different SVG. Rinse, repeat.

I'm saying an example of what not to do is still an example.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#186

Post by wewewedxfgdf »

The pelican benchmark is exactly what's wrong with hiring in technology.

It's got nothing to do with what most people actually do when they're working - just like most job interviews which ask you to draw a pelican as their way of assessing you.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#187

Post by huxley »

wewewedxfgdf wrote: post #186

The pelican benchmark is exactly what's wrong with hiring in technology. It's got nothing to do with what most people actually do when they're working - just like most job interviews which ask you to draw a pelican as their way of assessing you.

Exactly what someone without nine years of 10X pelican drawing experience would say

Re: Kimi K3, and what we can still learn from the pelican benchmark

#188

Post by duckerduck »

Xx_crazy420_xX wrote: post #5

I would be surprised if pelican svgs are not part of the training corpus rn

As mentioned elsewhere, the benchmark introduces bad pelicans in the training set. What I'm curious about however, if it's possible for a human artist to "poison" the benchmark by releasing some really good pelicans svgs and have all future models output their version.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#189

Post by embedding-shape »

michaelbuckbee wrote: post #63

Like Simon concludes the article, the main use of this isn't to say which model is "better", but to try and poke at the model to sort out things like quality vs cost vs speed. So I put together a quick comparison of the last couple iterations of Opus, Fable and now Kimi. Kimi is cheapest by 5x but also slowest by 2x https://9gpyw4uxr2.evvl.io/

Personally I'd consider the three middle ones to be failing, in the typical "Gemini/Google" fashion in that the model is doing more than what the prompt asks. The prompt asks for SVG, yet the model is providing more.

Edit: Actually, looking at the K2.6 response, that's borderline failing too, it's using HTML+CSS+SVG, not just SVG, again failing to follow the prompt properly.

By the way, that website seems like a black hole for information, it says "Expires in 6 days" in the top right which seems really weird for a page hosting couple of KB of data at most.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#190

Post by onion2k »

wewewedxfgdf wrote: post #186

The pelican benchmark is exactly what's wrong with hiring in technology. It's got nothing to do with what most people actually do when they're working - just like most job interviews which ask you to draw a pelican as their way of assessing you.

It's got nothing to do with what most people actually do when they're working..

AI companies claim their products are generalists though, and that they can do a good job on anything you give them, so you can't say what people will be doing with it. "Generate an SVG of an bird on a bicycle" is a corner case certainly but if a candidate interviewing for a role claims they can handle the corner cases then it's totally fair to assess them on that.

Besides, if you move up one layer to "how good is AI at generating valid SVG markup of non-obvious things", pelican on a bike is actually a good test.

Post reply on HN