Huh. They're not "Pelicanmaxxing"... they're Ottermaxxing. Take a look at the GLM 5.2 and Deepseek V4 "animal on a plane" examples. In every case, the animal is standing on top of the plane, a clear misunderstanding of the concept of "animal on a plane"... with the exception of the Otter. The otters are sitting in a seat on the plane, looking out the window. That's Ethan Mollick's "Otter On A Plane Using WiFi" image…
Thank you for the share, I'm glad AI labs are getting their priorities straight.
Are AI labs pelicanmaxxing?
231–240 of 253 posts
Re: Are AI labs pelicanmaxxing?
#232This is fantastic I've been casually spot-checking other animals in other vehicles, because my absolute dream situation here is to catch an AI lab that's demonstrably better at pelicans on bicycles than other combinations. Catching a lab cheating specifically on my one dumb benchmark would be really funny . Dylan's methodology here - generating 1008 SVGs across an 8x6 combination - is significantly more robust than a…
Re: Are AI labs pelicanmaxxing?
#233Earlier quoted context omitted.
Honest question how could they possibly train on this as there are no good SVG pelicans to train off right? So they’re just training off a bunch of bad ones which should lead to just bad pelicans, but the pelicans are getting better.
It’s not hard for a visual model to score the quality of that output though, which would be a pretty good fitness function.
Re: Are AI labs pelicanmaxxing?
#234Re: Are AI labs pelicanmaxxing?
#235Earlier quoted context omitted.
These are rectangles with some fill and strokes, textboxes lines and arrows. If the text has already been written, and the author has a clear idea of the flow all it takes are some a dozen or so points and click, and setting a couple of fills, then copy pastes, and finally some resizes, selects and moves. This design is trivial. I admit it would be hard to achieve in Word (or at least for me because I don’t know how…
> I’m sure they will end up with a fun little toy from the whole endeavor they can play with for 2 weeks before abandoning. That's exactly the idea - except it's more like 2 hours before I prototype the next version. > Maybe the author will even feel bad about the carbon footprint of this whole exercise and buy some carbon offsets to make up for it. The passive aggressiveness of this is perfectly weighted and admirab…
Re: Are AI labs pelicanmaxxing?
#236However thinking about it...if someone asked me to manually create an SVG, or hell even draw a quick doodle on a bit of paper of the same, I'd still probably be much slower than an LLM and potentially end up sketching less accurate anatomy than the machine.
I think the general "organic task" stuff has been mostly sorted out, but in personal and professional experiences using AI to try to _do_ something, I've found less so recently problems with hallucinations and moreso problems with attention.
For example GPT5.6 still has issues where if I provide it with a list of documents and then ask it to raise questions from that information. Then provide it with additional documents that answer some of those questions and ask it to summarise which outstanding questions there are again, it still asks questions that have become irrelevant with the additional documents - but when this is pointed out it knows exactly what to do and produces the correct list of outstanding questions.
I'm sure frontier models are doing all sorts of crazy stuff with attention already, but it seems to me like we almost need some hierarchical attention mechanism like KVL (with Level added) so that it's aware not only of semantic connections between tokens in the context but also of where there are gaps, missing links to assist the model in becoming aware of its own attention span (I guess).
Re: Are AI labs pelicanmaxxing?
#237Re: Are AI labs pelicanmaxxing?
#238I'm glad someone ran the numbers on this. Every single Simon Willison post of an SVG is followed with someone dismissing it saying "I'm sure they train on it by now." This is despite a good blog post with sound logic on how easy that is to catch. [1] Glad to see someone took the time for a quantitative analysis of dumb little animals riding dumb little bikes. 1. https://simonwillison.net/2025/Nov/13/training-for-peli…
> Using a single LLM judge for scoring. Every score here comes from one model, GPT-5.6 Luna, looking at one image at a time. I didn’t do much alignment and didn’t check how often it agrees with itself on a re-run.
Having used a similar setup (with previous gen LLMs) to evaluate the 3D models that my product[0] generates, it turned out there was no correlation at all. LLM judgments were very much random and I assume judging SVGs is not that far from judging 3D models. I guess I have to re-test this with current gen.
Re: Are AI labs pelicanmaxxing?
#239Earlier quoted context omitted.
I'm about ready for maxxmaxxing, where there's just maximally more of everything all the time
I’m waiting for the inevitable counterreaction to all this maxxing business, dubbed "minmaxxing"
Minimizing peaks is probably not a good strategy generally. Even low peaks may have some benefit ("if you know your problem is in this domain and performance is critical, this tool offers a 3% advantage").
Minimizing troughs may be more attractive. People generally seem to react more strongly to negatives and competitors can devise benchmarks which emphasize one's troughs.
While a universal expert would be convenient and broad knowledge aids some forms of creativity, specialization has substantial advantages.
Trading max performance (primary metric of concern) for some improvement in a secondary metric often makes sense and that could be an interpretation of "minmaxxing", reducing the over-emphasis of a single metric which would otherwise be maximized.
Re: Are AI labs pelicanmaxxing?
#240> But again, some combinations might be just harder to draw than others.
> To account for that, I fit a fixed-effects regression on all 1,008 images: score ~ lab + animal × vehicle, plus per-lab interaction terms for pelican, bicycle, and the pelican-bicycle cell, with robust standard errors. The animal × vehicle terms absorb the inherent difficulty of all 48 combinations. The interactions measure each lab’s benchmark-specific boost relative to the average lab, with confidence intervals.