Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are AI labs pelicanmaxxing?

dylancastillo.co

Post reply on HN
Print view View original

101–110 of 253 posts

Re: Are AI labs pelicanmaxxing?

#101

Post by dllu »

cherioo wrote: post #83
dllu wrote: post #24

I feel like getting LLMs to spit out an SVG is akin to getting a human artist to draw something by just reciting a list of coordinates. It's insanely hard and unnatural. Image generation models nowadays can easily generate a photorealistic pelican riding a bicycle, where the bicycle has perfect structure. But it is, of course, only a raster image. It seems that we're missing a kind of step to decompose an image into…

I don’t quite agree. Good human artist can visualize in their mind how to draw a picture, i think. Which i think is no different than LLM doing SVG drawing in their “head”. Anthropic’s recent post call this head-space “workspace”. It just might feel foreign to human who does not have a SVG trained head-space.

Sorry, my comment was confusingly worded, but I am precisely thinking about the head-space thing you are talking about. My point is: the fact that image generation models can generate a perfect pelican riding on a topologically correct and highly detailed bike frame [1], suggests that AI does have the capability to correctly understand it internally. But something is disconnected when we just let an LLM directly output the SVG, and all the nuanced understanding is lost.

As such, modern LLMs still kind of suck at generating pelican bike SVGs with obvious errors:

* some omitted the bottom of the diamond which connects from the pedals to the rear wheel

* some added an extra connection from the pedals to the front wheel, making it impossible to steer

* none could align the head tube with the fork

* none added a correct offset to the fork

* none could generate the chain properly in a way that attaches to the two sprockets correctly

... whereas these errors do not appear in the raster image. (To be fair, the raster image has other weirdnesses, like the bird having arms and only one leg)

If we could harness the sort of internal thinking that must have happened when it generated the highly consistent raster image, but make it output SVG instead, we'd get much better SVGs. So this would be indeed "visualize in their mind how to draw a picture" before outputting the SVG.

[1] https://chatgpt.com/s/m_6a611c29c02481918fbb0f165eb83594

Re: Are AI labs pelicanmaxxing?

#102

Post by OutOfHere »

qq66 wrote: post #59
lukev wrote: post #44

Earlier quoted context omitted.

What if they’re not pelicanmaxxing, but svgmaxxxing in general? Because otherwise using a LLM to generate complex svgs is pretty niche and what I thought made this a good benchmark when it was new - generalized programming and spatial knowledge. Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.

But that's a genuine worthwhile capability. It's like benchmarkmaxxing on a weightlifting competition by getting really strong.

The fear is that the SVGmaxxing is limited to "X doing Y". If such 'template maxxing' exists, it will break for other templates, e.g. "X not doing Y", "X and Y doing Z", "X doing Y doing Z", etc.

Re: Are AI labs pelicanmaxxing?

#103

Post by Topfi »

apwheele wrote: post #23

So this is not my experience at all for asking about simple SVG icons for web-pages. Here is one of the examples I have tried for in the past, make a simple cartoon SVG knife for a map icon for a crime map. https://x.com/CrimeDecoder/status/2080008114615537766 Can see the images for ChatGPT/Claude (Sonnet 5), and Gemini are all quite bad. Jagged edge of LLMs. How do you explain being able to generate very complicated…

Thanks for sharing another solid data point. I fear you won't get an answer from my experience [0]. Unfortunately, the blog post decided to forgo the very models that I found to be the worst offenders:

> Here is what Qwen3.6-35B-A3B via Openrouter provided for a sloth riding a skateboard: https://imgur.com/a/Dy8fvR5

> Like Grok 4 Fasts attempt at a mushroom in a rowboat, it is barely recognisable as anything despite both Qwen3.6-35B-A3B and Grok 4 Fast having no issue with more popular (i.e. benchmarked) examples. [...]

> And here is Opus 4.7 [which simonw claimed to provide a worse pelican vs Qwen], again via Openrouter: https://imgur.com/a/Qus1Enf

Anyone who hasn't witnessed such deltas either hasn't looked at enough examples, a sufficient variety of models, or both. And they are, unfortunately, not limited to "SVGMaxxing", but a wide range of evals.

[0] https://news.ycombinator.com/item?id=48951229

Re: Are AI labs pelicanmaxxing?

#105

Post by conception »

unholiness wrote: post #51
stusmall wrote: post #10

I'm glad someone ran the numbers on this. Every single Simon Willison post of an SVG is followed with someone dismissing it saying "I'm sure they train on it by now." This is despite a good blog post with sound logic on how easy that is to catch. [1] Glad to see someone took the time for a quantitative analysis of dumb little animals riding dumb little bikes. 1. https://simonwillison.net/2025/Nov/13/training-for-peli…

I don't think this small amount generalization to other animals and vehicles is strong evidence they haven't trained on this, either directly or more generally.

Honest question how could they possibly train on this as there are no good SVG pelicans to train off right? So they’re just training off a bunch of bad ones which should lead to just bad pelicans, but the pelicans are getting better.

Re: Are AI labs pelicanmaxxing?

#107

Post by michaelt »

bnfcl wrote: post #39

This is funny, I actually did a similar experiment just yesterday. Looking for evidence of the same, but with another twist: checking if the models would choose to create a pelican on a bicycle, if no specific bird or method of transportation was specified. My version of it: https://www.modelbias.ai/pelican-on-a-bicycle-test

The outputs of Qwen3.7 Plus have to be seen to be believed:

https://www.modelbias.ai/pelican-on-a-bicycle-test/result/12... https://www.modelbias.ai/pelican-on-a-bicycle-test/result/12...

Re: Are AI labs pelicanmaxxing?

#108

Post by johndough »

ACCount37 wrote: post #13
johndough wrote: post #4

Another point for consideration: Specialized SVG models create way better looking pelicans riding a bicycle. (E.g. Refract V4: https://jumpshare.com/s/8liB7Aiuoo3yucbWGXjZ mirror: https://postimg.cc/McV70p84 )

The name is "Recraft V4", and from looking it up: yeah, it sure seems like whatever black magic they use for SVG generation kicks ass.

Oops, autocorrect. Sorry about that.

Re: Are AI labs pelicanmaxxing?

#109

Post by johndough »

solarkraft wrote: post #8
johndough wrote: post #4

Another point for consideration: Specialized SVG models create way better looking pelicans riding a bicycle. (E.g. Refract V4: https://jumpshare.com/s/8liB7Aiuoo3yucbWGXjZ mirror: https://postimg.cc/McV70p84 )

That’s an impressive image, but what a mistake it was to click the second link (on mobile without an ad blocker). I wouldn’t send it to anyone I respect ...

Thanks for pointing that out. I haven't noticed any adds in years with Firefox and Ublock Origin extension.

I'll look for a better image host in the future. I guess the economic incentives makes them all turn bad after a while.

Re: Are AI labs pelicanmaxxing?

#110

Post by waterproof »

I recently had the following conversation with Claude:

Me: how many P's are in the following text? [Pasted text]

Claude: There are 14 P's, all lowercase (no capital P's)

Me: how many in "strawberry"?

Claude: there are 3 R's in the word "strawberry".

Post reply on HN