Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K3, and what we can still learn from the pelican benchmark

simonwillison.net

Post reply on HN
Print view View original

111–120 of 245 posts

Re: Kimi K3, and what we can still learn from the pelican benchmark

#111

Post by [deleted] »

Topfi wrote: post #87
simonw wrote: post #68

Earlier quoted context omitted.

> I have shared examples of certain models by certain labs doing far better on the pelican cycling vs other, similar prompts Please share those again! One of the things I'm most looking forward to is a lab producing a model that creates a really great pelican riding a bicycle and then a terrible sloth riding a skateboard (or whatever). I've not seen that myself yet.

Happy to, here one example where Grok 4 Fast, despite producing a fairly consistent pelican [0], did severely worse in a similarly outlandish scenario along with Haiku 4.5 and GPT-5 for context: https://news.ycombinator.com/item?id=45599403 > [...] a really great pelican riding a bicycle and then a terrible sloth riding a skateboard [...] Happy to play ball. You made a blog post a few weeks back on one of the Qwen mo…

[deleted]

Re: Kimi K3, and what we can still learn from the pelican benchmark

#112

Post by forgot-my-pw »

simonw wrote: post #80
OtherShrezzing wrote: post #75

Earlier quoted context omitted.

Respectfully, the pelicans used to be an unrecognisable mess and now they’re unquestionably pelicans on bicycles, rendered poorly, from every model. In the same timescale, model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts. Moreover, they have a uniform style, even though your prompt doesn’t ask for one. There's no model going rogue and pro…

You know what, that's actually something I hadn't considered before. There's definitely a bias towards a pelican cycling from left to right on a red bicycle against a blue sky and green grass. Blue sky and green grass aren't that surprising, but the color and direction are interesting. When I finally build the proper gallery I'll throw in a few other creature-vehicle combinations, and track some characteristics like…

I have done some variation of the other animals, also for something more tricky where they need to calculate things, I ask them to draw an SVG at a certain angle.

For example: "generate an SVG of a chessboard seen from a 45 degree angle slightly higher POV" or "generate an SVG of a basketball court from a TV broadcast perspective".

I find Gemini is still the best at creating SVGs.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#113

Post by OtomotO »

OsrsNeedsf2P wrote: post #13

It's incredible Simon still believes pelicans on bikes aren't part of the training set, despite hundreds of them on blogs, forums, and Github. Stuff we put in our company blog shows up known by LLMs 6 months later, and we have 1000x less traffic than Simon's own website

It's incredible people still discuss the pelicans... But then again, the ad just works.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#114

Post by nick3000 »

pterhx wrote: post #90
simonw wrote: post #80

Earlier quoted context omitted.

You know what, that's actually something I hadn't considered before. There's definitely a bias towards a pelican cycling from left to right on a red bicycle against a blue sky and green grass. Blue sky and green grass aren't that surprising, but the color and direction are interesting. When I finally build the proper gallery I'll throw in a few other creature-vehicle combinations, and track some characteristics like…

In photography (and probably art in general), there's a composition "rule" to frame moving subjects from left to right. So the direction may not be that interesting!

The other thing to consider (as someone who frequently take a photos of their bike) the common direction has the drive side out! In cycling forums it is sacrilegious to post a photo of your bicycle without showing the drive side.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#115

Post by lIl-IIIl »

simonw wrote: post #80
OtherShrezzing wrote: post #75

Earlier quoted context omitted.

Respectfully, the pelicans used to be an unrecognisable mess and now they’re unquestionably pelicans on bicycles, rendered poorly, from every model. In the same timescale, model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts. Moreover, they have a uniform style, even though your prompt doesn’t ask for one. There's no model going rogue and pro…

You know what, that's actually something I hadn't considered before. There's definitely a bias towards a pelican cycling from left to right on a red bicycle against a blue sky and green grass. Blue sky and green grass aren't that surprising, but the color and direction are interesting. When I finally build the proper gallery I'll throw in a few other creature-vehicle combinations, and track some characteristics like…

Bicycle color, grass color and sky color are all part of the prompt.

>Cartoon illustration of a white pelican wearing a red scarf, riding a red bicycle along a gray road with white dashed lines; the pelican has a large orange beak and webbed orange feet pedaling, with white motion lines behind it; the background shows a light blue sky with white clouds, a yellow sun, two small black birds in flight, and green grass with tiny white flowers in the foreground

Re: Kimi K3, and what we can still learn from the pelican benchmark

#116

Post by komadori »

lIl-IIIl wrote: post #115
simonw wrote: post #80

Earlier quoted context omitted.

You know what, that's actually something I hadn't considered before. There's definitely a bias towards a pelican cycling from left to right on a red bicycle against a blue sky and green grass. Blue sky and green grass aren't that surprising, but the color and direction are interesting. When I finally build the proper gallery I'll throw in a few other creature-vehicle combinations, and track some characteristics like…

Bicycle color, grass color and sky color are all part of the prompt. >Cartoon illustration of a white pelican wearing a red scarf, riding a red bicycle along a gray road with white dashed lines; the pelican has a large orange beak and webbed orange feet pedaling, with white motion lines behind it; the background shows a light blue sky with white clouds, a yellow sun, two small black birds in flight, and green grass w…

That wasn't the prompt. That text was generated by asking the model to describe an image and feeding it a rendering of the SVG it had previously generated.

Re: Kimi K3, and what we can still learn from the pelican benchmark

#117

Post by danielrmay »

simonw wrote: post #66
OsrsNeedsf2P wrote: post #13

It's incredible Simon still believes pelicans on bikes aren't part of the training set, despite hundreds of them on blogs, forums, and Github. Stuff we put in our company blog shows up known by LLMs 6 months later, and we have 1000x less traffic than Simon's own website

The pelicans are still all rubbish. If they make it into the training set it doesn't help the models produce better pelicans, if anything it will make them perform worse!

What does good look like?

Re: Kimi K3, and what we can still learn from the pelican benchmark

#118

Post by filoleg »

ahtihn wrote: post #94
pterhx wrote: post #90

Earlier quoted context omitted.

In photography (and probably art in general), there's a composition "rule" to frame moving subjects from left to right. So the direction may not be that interesting!

I wonder if that changes in countries where the main language is written right to left?

That was my first thought too, I wonder if it works the same in countries speaking arabic (as that's the first one i could think of that's a language with truly no-buts right to left writing).

Re: Kimi K3, and what we can still learn from the pelican benchmark

#119

Post by simonw »

lIl-IIIl wrote: post #115
simonw wrote: post #80

Earlier quoted context omitted.

You know what, that's actually something I hadn't considered before. There's definitely a bias towards a pelican cycling from left to right on a red bicycle against a blue sky and green grass. Blue sky and green grass aren't that surprising, but the color and direction are interesting. When I finally build the proper gallery I'll throw in a few other creature-vehicle combinations, and track some characteristics like…

Bicycle color, grass color and sky color are all part of the prompt. >Cartoon illustration of a white pelican wearing a red scarf, riding a red bicycle along a gray road with white dashed lines; the pelican has a large orange beak and webbed orange feet pedaling, with white motion lines behind it; the background shows a light blue sky with white clouds, a yellow sun, two small black birds in flight, and green grass w…

No, the prompt I always use is "Generate an SVG of a pelican riding a bicycle".

Re: Kimi K3, and what we can still learn from the pelican benchmark

#120

Post by simonw »

danielrmay wrote: post #117
simonw wrote: post #66

Earlier quoted context omitted.

The pelicans are still all rubbish. If they make it into the training set it doesn't help the models produce better pelicans, if anything it will make them perform worse!

What does good look like?

The dedicated text-to-image models all produce good illustrations of pelicans riding bicycles. Here's one I got from OpenAI's gpt-image-2 just the other day: https://simonwillison.net/2026/Jul/14/pedalican/
Post reply on HN