Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 3 Deep Think drew me a good SVG of a pelican riding a bicycle

simonwillison.net

Post reply on HN
Print view View original

31–40 of 63 posts

Re: Gemini 3 Deep Think drew me a good SVG of a pelican riding a bicycle

#31

Post by romanhn »

alterom wrote: post #27
vessenes wrote: post #9

Simon notes this benchmark is win-win, since he loves pictures of pelicans riding bicycles — if they spend time benchmaxxing it’s like free pelicans for him. He originally promised to generate a bunch more animals when we got a “good” pelican. This is not a good pelican. This is an OUTSTANDING pelican, a great bicycle, and it even has a little sun ray over the ocean marked out. I’d like to see more animals please Sim…

> a great bicycle It's not. Sorry. Go look at some real bicycles for reference.

A better comparison would be the monstrosities generated by older models.

Re: Gemini 3 Deep Think drew me a good SVG of a pelican riding a bicycle

#32

Post by Springtime »

I have wondered if with these tests it'll reach a point where online models cheat by generating a line art raster reference then behind the scenes deciding how to vectorize it in the most minimalist way (eg: using strokes and shape elements, etc, rather than naively using path outlines for all forms).

Re: Gemini 3 Deep Think drew me a good SVG of a pelican riding a bicycle

#34

Post by bonesss »

rustyhancock wrote: post #2

The intensity of competition between models is so intense right now they are definitely benchmaxxing pelican on bike SVGs and Will Smith spaghetti dinner videos.

Parallel hypothesis: the intensity of competition between models is so intense that any high-engagement high-relevance web discussion about any LLM/AI generation is gonna hit the self-guided self-reinforced model training and result in de facto benchmaxxing.

Which is only to say: if we HN-front-page it, they will come (generate).

Re: Gemini 3 Deep Think drew me a good SVG of a pelican riding a bicycle

#35

Post by taberiand »

Springtime wrote: post #32

I have wondered if with these tests it'll reach a point where online models cheat by generating a line art raster reference then behind the scenes deciding how to vectorize it in the most minimalist way (eg: using strokes and shape elements, etc, rather than naively using path outlines for all forms).

Is that cheating, or is that just working smarter not harder?

Re: Gemini 3 Deep Think drew me a good SVG of a pelican riding a bicycle

#36

Post by alestainer »

Interesting thing: I've got my internal request that is similar to this pelican. And there was 0 progress on it in the past ~2 years. Which might have at least a couple of explanations. 1. Spillage into the pre-training: some real artist had drawn a pelican riding a bicycle. 2. Seeing it as an important discourse for model intelligence in the training data might affect allocation of compute into solving this problem, either thru engineers or the model itself finding the texts about this challenge.

Re: Gemini 3 Deep Think drew me a good SVG of a pelican riding a bicycle

#37

Post by sdenton4 »

alterom wrote: post #27
vessenes wrote: post #9

Simon notes this benchmark is win-win, since he loves pictures of pelicans riding bicycles — if they spend time benchmaxxing it’s like free pelicans for him. He originally promised to generate a bunch more animals when we got a “good” pelican. This is not a good pelican. This is an OUTSTANDING pelican, a great bicycle, and it even has a little sun ray over the ocean marked out. I’d like to see more animals please Sim…

> a great bicycle It's not. Sorry. Go look at some real bicycles for reference.

This is a very reasonable drawing of a bicycle. It has a solid rear triangle, and forward swept front fork, which is an important detail for actually being able to steer the bike. The drivetrain is single speed, but that's fine, and the wheels are radially laced, which is also fine: both of those simplified details are things which occur in real bicycles.

Re: Gemini 3 Deep Think drew me a good SVG of a pelican riding a bicycle

#38

Post by WarmWash »

segmondy wrote: post #18

For those claiming they rigged it. Do you have any concrete evidence? What if the models have just gotten really good? I just asked Gemini pro to generate an SVG of an octopus dunking a basketball and it did a great job. Not even Deep Think model. Then I did "generate an svg of raccoon at a beach drinking a beer" you can go try this out yourself. Ask it to generate anything you want in SVG. use your imagination. Rant…

Simon has a private set of SVG tests he uses as well. He said that the private ones were just as impressive.

Re: Gemini 3 Deep Think drew me a good SVG of a pelican riding a bicycle

#39

Post by WarmWash »

Are AI labs training on the bike Pelican?

From the blog:

>The strongest argument is that they would get caught. If a model finally comes out that produces an excellent SVG of a pelican riding a bicycle you can bet I’m going to test it on all manner of creatures riding all sorts of transportation devices. If those are notably worse it’s going to be pretty obvious what happened.

He mentioned in the Deep Think thread the other day that his secret test set also was impressive.

Re: Gemini 3 Deep Think drew me a good SVG of a pelican riding a bicycle

#40

Post by Springtime »

taberiand wrote: post #35
Springtime wrote: post #32

I have wondered if with these tests it'll reach a point where online models cheat by generating a line art raster reference then behind the scenes deciding how to vectorize it in the most minimalist way (eg: using strokes and shape elements, etc, rather than naively using path outlines for all forms).

Is that cheating, or is that just working smarter not harder?

The interesting aspect of the ongoing tests I feel is seeing how models can plan out an image directly using SVG primatives solely through reasoning (code-to-code). If they have a reference then it's a different type of challenge (optimizing for a trace).
Post reply on HN