Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are AI labs pelicanmaxxing?

dylancastillo.co

Post reply on HN
Print view View original

51–60 of 253 posts

Re: Are AI labs pelicanmaxxing?

#51

Post by unholiness »

stusmall wrote: post #10

I'm glad someone ran the numbers on this. Every single Simon Willison post of an SVG is followed with someone dismissing it saying "I'm sure they train on it by now." This is despite a good blog post with sound logic on how easy that is to catch. [1] Glad to see someone took the time for a quantitative analysis of dumb little animals riding dumb little bikes. 1. https://simonwillison.net/2025/Nov/13/training-for-peli…

I don't think this small amount generalization to other animals and vehicles is strong evidence they haven't trained on this, either directly or more generally.

Re: Are AI labs pelicanmaxxing?

#53

Post by ramses0 »

Rooster61 wrote: post #18

I find it humorous that the animal + plane combo appears to be such an outlier. I assume this is due to the models assuming the user mean plain and misspelled it in the prompt.

I think it's actually due to "pelican on a plane" isn't the same as "pelican on an airplane" (Sonnet5 @ Flamingo x Plane), some consistent and warranted semantic/linguistic confusion!

Re: Are AI labs pelicanmaxxing?

#54

Post by gbalduzzi »

ck2 wrote: post #49

I am not sure if this is how it works but let's say there was a reddit thread talking about the pelican benchmark and in it someone posts mockup examples of what an ideal result would look like aren't some LLM going to digest that thread at some point and indirectly learn from it? basically my point is originally this was a good benchmark because it was an absurd never-seen-before thing, but now that it is in content…

They ingest so much data that a couple of reddit threads do not move the needle.

It is the reinforcement learning that produces more tangible results with less data, but it is something that the AI labs specifically selects and it is not picked up unknowingly

Re: Are AI labs pelicanmaxxing?

#56

Post by lukev »

netsec_burn wrote: post #50
lukev wrote: post #44

Earlier quoted context omitted.

What if they’re not pelicanmaxxing, but svgmaxxxing in general? Because otherwise using a LLM to generate complex svgs is pretty niche and what I thought made this a good benchmark when it was new - generalized programming and spatial knowledge. Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.

Addressed in the article, in case you're curious.

Well, it’s mentioned as a limitation of the analysis, very much not ruled out (or in.)

That simonw is causing labs to do extra fine-tuning runs for this seems highly probable :)

Re: Are AI labs pelicanmaxxing?

#57

Post by BigTTYGothGF »

andrewstuart wrote: post #34

The pelican prompt is ridiculous. Test the LLLM against things you want it to do. Asking questions that are absurd is like interviewing developers and asking absurd questions on the grounds that it tests creative and critical thinking. Remember these Microsoft interview questions designed to identify the best developers? "If you could eliminate one U.S. state, which one would it be?" "How would you move Mount Fuji?"…

> Test the LLLM against things you want it to do

I agree, it is ridiculous to ask an LLM to replace an artist.

Re: Are AI labs pelicanmaxxing?

#58

Post by munk-a »

It's a method to grade LLM output - as such it's something that will receive focus in correcting for. As soon as people who have a say in where funding is going noticed it as a metric the labs started caring about their performance in it. In the best case the labs are focusing on improving SVG capabilities in general and optimizing Pelican production as part of that initiative - but now that it's a known measure it is no longer reliable.

Re: Are AI labs pelicanmaxxing?

#59

Post by qq66 »

lukev wrote: post #44
simonw wrote: post #17

This is fantastic I've been casually spot-checking other animals in other vehicles, because my absolute dream situation here is to catch an AI lab that's demonstrably better at pelicans on bicycles than other combinations. Catching a lab cheating specifically on my one dumb benchmark would be really funny . Dylan's methodology here - generating 1008 SVGs across an 8x6 combination - is significantly more robust than a…

What if they’re not pelicanmaxxing, but svgmaxxxing in general? Because otherwise using a LLM to generate complex svgs is pretty niche and what I thought made this a good benchmark when it was new - generalized programming and spatial knowledge. Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.

But that's a genuine worthwhile capability. It's like benchmarkmaxxing on a weightlifting competition by getting really strong.

Re: Are AI labs pelicanmaxxing?

#60

Post by dbt00 »

beering wrote: post #16
Wowfunhappy wrote: post #7

> The more plausible story is SVGmaxxing Exactly--and you have to ask yourself at this point what "maxxing" really means, since "get better at drawing SVGs" is a useful skill.

Really awful how the AI labs are skillmaxxing /s Pelicans aside, we need to remember that benchmarks are the only good quantitative way we have of comparing models. If someone has complaints about “benchmaxxing”, please ask them to contribute a better benchmark! It is valuable work and very appreciated.

It's a problem because of Goodhart's law.

If you train towards the test, you aren't necessarily improving overall fitness, but you are destroying the value of that test over time because you're decreasing its correlation with overall fitness.

Post reply on HN