Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are AI labs pelicanmaxxing?

dylancastillo.co

Post reply on HN
Print view View original

71–80 of 253 posts

Re: Are AI labs pelicanmaxxing?

#71

Post by zahlman »

ryukoposting wrote: post #63
qq66 wrote: post #59

Earlier quoted context omitted.

But that's a genuine worthwhile capability. It's like benchmarkmaxxing on a weightlifting competition by getting really strong.

How useful actually is this? It generates SVGs of pelicans on bicycles, sure, and some of them are (almost) spatially correct. But, none of them look good . AI image generation suffers from this more generally. You can generate pictures of pelicans, sure. Newer models clearly generate images with more pelican-ness than before. But all of it is still uglier than sin. Drawing things accurately is one thing, making resu…

Even if the models don't break through any particular "uglier than sin" barrier, with a bit more work, presumably the SVGs could become importable into an editor that would let a human apply taste and discretion. Seems to me like a heck of a head-start.

As for conventional diffusion-model stuff, I happen to think there are some pieces of AI art that still look really good even knowing they're AI.

Re: Are AI labs pelicanmaxxing?

#72

Post by wasabi991011 »

bnfcl wrote: post #39

This is funny, I actually did a similar experiment just yesterday. Looking for evidence of the same, but with another twist: checking if the models would choose to create a pelican on a bicycle, if no specific bird or method of transportation was specified. My version of it: https://www.modelbias.ai/pelican-on-a-bicycle-test

I find your analysis much more convincing than TFA, since it doesn't require a subjective evaluation and is more robust to animal/transport complexity.

Re: Are AI labs pelicanmaxxing?

#73

Post by flsw »

Rooster61 wrote: post #18

I find it humorous that the animal + plane combo appears to be such an outlier. I assume this is due to the models assuming the user mean plain and misspelled it in the prompt.

I noticed this happens especially with herons. My guess is it's because the model links "heron" to Heron's formula and the Cartesian plane

Re: Are AI labs pelicanmaxxing?

#74

Post by zahlman »

docheinestages wrote: post #62
simonw wrote: post #17

This is fantastic I've been casually spot-checking other animals in other vehicles, because my absolute dream situation here is to catch an AI lab that's demonstrably better at pelicans on bicycles than other combinations. Catching a lab cheating specifically on my one dumb benchmark would be really funny . Dylan's methodology here - generating 1008 SVGs across an 8x6 combination - is significantly more robust than a…

I think a more fundamental test is SVG art creation in general. Perhaps a pipeline to take any image, caption it, ask the LLM for an SVG, rasterize to an image, and finally either use a deterministic visual similarity check or ask another LLM to be the judge and score how close the SVG is to the original image.

Fidelity to the original is definitely not how humans would measure "art" in this context.

Re: Are AI labs pelicanmaxxing?

#77

Post by amarant »

ryukoposting wrote: post #63
qq66 wrote: post #59

Earlier quoted context omitted.

But that's a genuine worthwhile capability. It's like benchmarkmaxxing on a weightlifting competition by getting really strong.

How useful actually is this? It generates SVGs of pelicans on bicycles, sure, and some of them are (almost) spatially correct. But, none of them look good . AI image generation suffers from this more generally. You can generate pictures of pelicans, sure. Newer models clearly generate images with more pelican-ness than before. But all of it is still uglier than sin. Drawing things accurately is one thing, making resu…

Vibecoding a SVG based metroidvania as we speak! This is gonna be lit!

Re: Are AI labs pelicanmaxxing?

#78

Post by elicash »

stusmall wrote: post #10

I'm glad someone ran the numbers on this. Every single Simon Willison post of an SVG is followed with someone dismissing it saying "I'm sure they train on it by now." This is despite a good blog post with sound logic on how easy that is to catch. [1] Glad to see someone took the time for a quantitative analysis of dumb little animals riding dumb little bikes. 1. https://simonwillison.net/2025/Nov/13/training-for-peli…

There's that version of the argument version, but there's also the softer version: that there used to be no training material of illustrated pelicans on bicycles, but now you have actual artistically talented individuals drawing it and that could improve the performance even though the AI labs are sucking it up no differently than everything else.

This post proves that hasn't happened yet, either. Although maybe the bad results posted online are being trained on and that explains the UNDER performance.

Re: Are AI labs pelicanmaxxing?

#80

Post by user- »

andrewstuart wrote: post #34

The pelican prompt is ridiculous. Test the LLLM against things you want it to do. Asking questions that are absurd is like interviewing developers and asking absurd questions on the grounds that it tests creative and critical thinking. Remember these Microsoft interview questions designed to identify the best developers? "If you could eliminate one U.S. state, which one would it be?" "How would you move Mount Fuji?"…

The whole point of "AI" is arbitrary task completion. Why isn't a SVG drawing relevant for that?
Post reply on HN