Are AI labs pelicanmaxxing? • Hacker News Board

Re: Are AI labs pelicanmaxxing?

#11

Post by cute_boi » Wed, Jul 22, 2026, 6:50 PM UTC

andy99 wrote: post #3

If an AI researcher was going to pelicanmaxx, they would almost certainly apply the augmentations mentioned in the article during training, e.g. randomly selecting animals and conveyances. You’d want a model that generalizes well, just sfting in that specific prompt would be pretty bush league for a frontier lab. I don’t have any reason to believe they are gaming the benchmark, just saying. I do find the idea of a da…

At this point, I think there are so many pelican images in the pretraining data that drawing a pelican no longer makes sense as a model evaluation task.

Re: Are AI labs pelicanmaxxing?

#12

Post by jonatron » Wed, Jul 22, 2026, 6:51 PM UTC

OK, so we've done animal_vehicle, how about new SVG ideas each time? I just tried "make an SVG of a man sitting in a chair at a computer behind a desk" which gives more interesting results than the animalVehicle test.

Re: Are AI labs pelicanmaxxing?

#13

Post by ACCount37 » Wed, Jul 22, 2026, 6:56 PM UTC

johndough wrote: post #4

Another point for consideration: Specialized SVG models create way better looking pelicans riding a bicycle. (E.g. Refract V4: https://jumpshare.com/s/8liB7Aiuoo3yucbWGXjZ mirror: https://postimg.cc/McV70p84 )

The name is "Recraft V4", and from looking it up: yeah, it sure seems like whatever black magic they use for SVG generation kicks ass.

Re: Are AI labs pelicanmaxxing?

#14

Post by javier123454321 » Wed, Jul 22, 2026, 6:57 PM UTC

tomas789 wrote: post #6

Having an objective score is quite difficult. Maybe it would be better to do a pairwise comparison and calculate ELO?

If you want to, go ahead, but it seems to me the author already exceeded the energy expenditure that this question warranted.

Re: Are AI labs pelicanmaxxing?

#15

Post by j45 » Wed, Jul 22, 2026, 6:57 PM UTC

The models definitely seem to pay attention to the tests.

Since the tests can be generally gamed with directing descriptions at it non-deterministically, there's a greater chance the questions solution can be found.

Of course, hopefully the models are instead adding patterns and types of questions as well and it makes the models more capable, but it may be limited in how it transfers to other types of questions in breadth or depth.

Re: Are AI labs pelicanmaxxing?

#16

Post by beering » Wed, Jul 22, 2026, 7:03 PM UTC

Wowfunhappy wrote: post #7

> The more plausible story is SVGmaxxing Exactly--and you have to ask yourself at this point what "maxxing" really means, since "get better at drawing SVGs" is a useful skill.

Really awful how the AI labs are skillmaxxing /s

Pelicans aside, we need to remember that benchmarks are the only good quantitative way we have of comparing models. If someone has complaints about “benchmaxxing”, please ask them to contribute a better benchmark! It is valuable work and very appreciated.

Re: Are AI labs pelicanmaxxing?

#17

Post by simonw » Wed, Jul 22, 2026, 7:04 PM UTC

This is fantastic

I've been casually spot-checking other animals in other vehicles, because my absolute dream situation here is to catch an AI lab that's demonstrably better at pelicans on bicycles than other combinations.

Catching a lab cheating specifically on my one dumb benchmark would be really funny.

Dylan's methodology here - generating 1008 SVGs across an 8x6 combination - is significantly more robust than anything I was considering.

His conclusion:

> Nothing jumped out at me. I couldn’t find a case where the pelican-bicycle images looked noticeably better than the rest of that model’s grid.

Re: Are AI labs pelicanmaxxing?

#18

Post by Rooster61 » Wed, Jul 22, 2026, 7:05 PM UTC

I find it humorous that the animal + plane combo appears to be such an outlier. I assume this is due to the models assuming the user mean plain and misspelled it in the prompt.

Re: Are AI labs pelicanmaxxing?

#19

Post by ninju » Wed, Jul 22, 2026, 7:05 PM UTC

jonatron wrote: post #12

OK, so we've done animal_vehicle, how about new SVG ideas each time? I just tried "make an SVG of a man sitting in a chair at a computer behind a desk" which gives more interesting results than the animalVehicle test.

There probably good set of images of that description already so it does exercise the inference capability of the model

Re: Are AI labs pelicanmaxxing?

#20

Post by gilleain » Wed, Jul 22, 2026, 7:09 PM UTC

simonw wrote: post #17

This is fantastic I've been casually spot-checking other animals in other vehicles, because my absolute dream situation here is to catch an AI lab that's demonstrably better at pelicans on bicycles than other combinations. Catching a lab cheating specifically on my one dumb benchmark would be really funny . Dylan's methodology here - generating 1008 SVGs across an 8x6 combination - is significantly more robust than a…

Perhaps also vary the bird? Wikipedia tells me pelicans are in the order _Pelecaniformes_ so shoebills or herons might do.