Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 3 Deep Think drew me a good SVG of a pelican riding a bicycle

simonwillison.net

Post reply on HN
Print view View original

41–50 of 63 posts

Re: Gemini 3 Deep Think drew me a good SVG of a pelican riding a bicycle

#41

Post by raincole »

segmondy wrote: post #18

For those claiming they rigged it. Do you have any concrete evidence? What if the models have just gotten really good? I just asked Gemini pro to generate an SVG of an octopus dunking a basketball and it did a great job. Not even Deep Think model. Then I did "generate an svg of raccoon at a beach drinking a beer" you can go try this out yourself. Ask it to generate anything you want in SVG. use your imagination. Rant…

Every bit of improvement on AI ability will have the corresponding denial phrase. Some people still think AI can't generate the correct number of fingers today.

Re: Gemini 3 Deep Think drew me a good SVG of a pelican riding a bicycle

#42

Post by irthomasthomas »

segmondy wrote: post #18

For those claiming they rigged it. Do you have any concrete evidence? What if the models have just gotten really good? I just asked Gemini pro to generate an SVG of an octopus dunking a basketball and it did a great job. Not even Deep Think model. Then I did "generate an svg of raccoon at a beach drinking a beer" you can go try this out yourself. Ask it to generate anything you want in SVG. use your imagination. Rant…

Why frame it as rigging? I assume they would teach the models to improve on tasks the public find interesting. Then we just have to come up with more challenges for it.

Re: Gemini 3 Deep Think drew me a good SVG of a pelican riding a bicycle

#43

Post by bayindirh »

segmondy wrote: post #18

For those claiming they rigged it. Do you have any concrete evidence? What if the models have just gotten really good? I just asked Gemini pro to generate an SVG of an octopus dunking a basketball and it did a great job. Not even Deep Think model. Then I did "generate an svg of raccoon at a beach drinking a beer" you can go try this out yourself. Ask it to generate anything you want in SVG. use your imagination. Rant…

> For those claiming they rigged it.

I don't think they "rigged" it, but might be given a bit more push on that part since it's going for a very long time now.

Another benchmark is going on at [0]. It's pretty interesting. A perfect scoring model "borks" in the next iteration, for example.

> Rant: This is why AI is going to take over, folks are not even trying the least.

It might be drawing things alright, at least some cases. I seldom use it when my hours long researches doesn't take me to the place I want, and guess what? AI can't go there, either. It hallucinates things, makes up stuff, etc. For a couple of things I asked, it managed to find a single reference, and it was the thing I was looking for, so it works rarely in my cases.

Rant: This is why people are delusional. They test the happy path and claims it knows all the paths, and then some.

[0]: https://clocks.brianmoore.com/

Re: Gemini 3 Deep Think drew me a good SVG of a pelican riding a bicycle

#44

Post by dw_arthur »

segmondy wrote: post #18

For those claiming they rigged it. Do you have any concrete evidence? What if the models have just gotten really good? I just asked Gemini pro to generate an SVG of an octopus dunking a basketball and it did a great job. Not even Deep Think model. Then I did "generate an svg of raccoon at a beach drinking a beer" you can go try this out yourself. Ask it to generate anything you want in SVG. use your imagination. Rant…

Everyone should have their own private evals for models. If I ask a question and a model flat out gets it wrong sometimes I will put it in my test questions bank.

Re: Gemini 3 Deep Think drew me a good SVG of a pelican riding a bicycle

#45

Post by krackers »

irthomasthomas wrote: post #42
segmondy wrote: post #18

For those claiming they rigged it. Do you have any concrete evidence? What if the models have just gotten really good? I just asked Gemini pro to generate an SVG of an octopus dunking a basketball and it did a great job. Not even Deep Think model. Then I did "generate an svg of raccoon at a beach drinking a beer" you can go try this out yourself. Ask it to generate anything you want in SVG. use your imagination. Rant…

Why frame it as rigging? I assume they would teach the models to improve on tasks the public find interesting. Then we just have to come up with more challenges for it.

It's not rigging—it's just RL.

Re: Gemini 3 Deep Think drew me a good SVG of a pelican riding a bicycle

#46

Post by halJordan »

raincole wrote: post #41
segmondy wrote: post #18

For those claiming they rigged it. Do you have any concrete evidence? What if the models have just gotten really good? I just asked Gemini pro to generate an SVG of an octopus dunking a basketball and it did a great job. Not even Deep Think model. Then I did "generate an svg of raccoon at a beach drinking a beer" you can go try this out yourself. Ask it to generate anything you want in SVG. use your imagination. Rant…

Every bit of improvement on AI ability will have the corresponding denial phrase. Some people still think AI can't generate the correct number of fingers today.

I love to hate it when someone unironically thinks asking an llm how many letters are in a word is a good test

Re: Gemini 3 Deep Think drew me a good SVG of a pelican riding a bicycle

#47

Post by ManuelKiessling »

hnuser123456 wrote: post #28
vessenes wrote: post #9

Simon notes this benchmark is win-win, since he loves pictures of pelicans riding bicycles — if they spend time benchmaxxing it’s like free pelicans for him. He originally promised to generate a bunch more animals when we got a “good” pelican. This is not a good pelican. This is an OUTSTANDING pelican, a great bicycle, and it even has a little sun ray over the ocean marked out. I’d like to see more animals please Sim…

It is visually outstanding. The only thing that sticks out to me is that the steering column bends out forwards towards the ground (negative trail), which would make it oversteer rather than self-stabilize. Interestingly there's a slight positive trail bend in the second one, though.

> the steering column bends out forwards towards the ground

It’s really fascinating in a way because I would have thought that this would be one of the least difficult parts of the challenge.

Re: Gemini 3 Deep Think drew me a good SVG of a pelican riding a bicycle

#48

Post by simonw »

Springtime wrote: post #32

I have wondered if with these tests it'll reach a point where online models cheat by generating a line art raster reference then behind the scenes deciding how to vectorize it in the most minimalist way (eg: using strokes and shape elements, etc, rather than naively using path outlines for all forms).

This Deep Think one was so good that I did get suspicious that maybe it was at least rendering the SVG to an image and then "looking" at the image and tweaking it over a few iterations.

But the reasoning trace doesn't hint at that and looks legit to me: https://gist.github.com/simonw/7e317ebb5cf8e75b2fcec4d0694a8...

I also asked Deep Think what tools it has access to and it has Python and Bash but no internet access, and as far as I can tell that environment doesn't have any libraries or tools installed that can render an SVG to an image format that it could view.

Re: Gemini 3 Deep Think drew me a good SVG of a pelican riding a bicycle

#49

Post by ej88 »

segmondy wrote: post #18

For those claiming they rigged it. Do you have any concrete evidence? What if the models have just gotten really good? I just asked Gemini pro to generate an SVG of an octopus dunking a basketball and it did a great job. Not even Deep Think model. Then I did "generate an svg of raccoon at a beach drinking a beer" you can go try this out yourself. Ask it to generate anything you want in SVG. use your imagination. Rant…

"not enough people are emotionally prepared for if it’s not a bubble"

Re: Gemini 3 Deep Think drew me a good SVG of a pelican riding a bicycle

#50

Post by Jerrrrrrrry »

halJordan wrote: post #46
raincole wrote: post #41

Earlier quoted context omitted.

Every bit of improvement on AI ability will have the corresponding denial phrase. Some people still think AI can't generate the correct number of fingers today.

I love to hate it when someone unironically thinks asking an llm how many letters are in a word is a good test

It is a good test now, for reasoning models.

It was a terrible test for pure tokenized models, because the logit that carries the carry digit during summation has a decent chance at getting lost.

SOTA models should reason to generate a function that returns the count of a given character, evaluate the function with tests, and use it for the output.

Post reply on HN