Are AI labs pelicanmaxxing?

#1

Post by dcastm » Wed, Jul 22, 2026, 5:17 PM UTC

Are AI labs pelicanmaxxing?
dylancastillo.co

Re: Are AI labs pelicanmaxxing?

#2

Post by dcchambers » Wed, Jul 22, 2026, 6:26 PM UTC

It's incredible that each model has it's own style that remains relatively consistent throughout all of the different generated examples.

Re: Are AI labs pelicanmaxxing?

#3

Post by andy99 » Wed, Jul 22, 2026, 6:26 PM UTC

If an AI researcher was going to pelicanmaxx, they would almost certainly apply the augmentations mentioned in the article during training, e.g. randomly selecting animals and conveyances. You’d want a model that generalizes well, just sfting in that specific prompt would be pretty bush league for a frontier lab.

I don’t have any reason to believe they are gaming the benchmark, just saying. I do find the idea of a data labeller having to generate thousands of svgs of different animals on different modes of transportation quite funny though.

Re: Are AI labs pelicanmaxxing?

#4

Post by johndough » Wed, Jul 22, 2026, 6:32 PM UTC

Another point for consideration: Specialized SVG models create way better looking pelicans riding a bicycle. (E.g. Refract V4: https://jumpshare.com/s/8liB7Aiuoo3yucbWGXjZ mirror: https://postimg.cc/McV70p84 )

Re: Are AI labs pelicanmaxxing?

#5

Post by sbseitz » Wed, Jul 22, 2026, 6:36 PM UTC

I wish I could downvote this for Pelicanmaxxing lmao.

Re: Are AI labs pelicanmaxxing?

#6

Post by tomas789 » Wed, Jul 22, 2026, 6:37 PM UTC

Having an objective score is quite difficult. Maybe it would be better to do a pairwise comparison and calculate ELO?

Re: Are AI labs pelicanmaxxing?

#7

Post by Wowfunhappy » Wed, Jul 22, 2026, 6:43 PM UTC

> The more plausible story is SVGmaxxing

Exactly--and you have to ask yourself at this point what "maxxing" really means, since "get better at drawing SVGs" is a useful skill.

Re: Are AI labs pelicanmaxxing?

#8

Post by solarkraft » Wed, Jul 22, 2026, 6:47 PM UTC

johndough wrote: post #4

Another point for consideration: Specialized SVG models create way better looking pelicans riding a bicycle. (E.g. Refract V4: https://jumpshare.com/s/8liB7Aiuoo3yucbWGXjZ mirror: https://postimg.cc/McV70p84 )

That’s an impressive image, but what a mistake it was to click the second link (on mobile without an ad blocker). I wouldn’t send it to anyone I respect ...

Re: Are AI labs pelicanmaxxing?

#9

Post by cute_boi » Wed, Jul 22, 2026, 6:48 PM UTC

https://playcode.io/blog/macbook-svg-benchmark

I think we should stop using pelican benchmark.

Re: Are AI labs pelicanmaxxing?

#10

Post by stusmall » Wed, Jul 22, 2026, 6:50 PM UTC

I'm glad someone ran the numbers on this. Every single Simon Willison post of an SVG is followed with someone dismissing it saying "I'm sure they train on it by now." This is despite a good blog post with sound logic on how easy that is to catch. [1] Glad to see someone took the time for a quantitative analysis of dumb little animals riding dumb little bikes.

1. https://simonwillison.net/2025/Nov/13/training-for-pelicans-...