Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are AI labs pelicanmaxxing?

dylancastillo.co

Post reply on HN
Print view View original

151–160 of 253 posts

Re: Are AI labs pelicanmaxxing?

#151

Post by nbardy »

lukev wrote: post #44
simonw wrote: post #17

This is fantastic I've been casually spot-checking other animals in other vehicles, because my absolute dream situation here is to catch an AI lab that's demonstrably better at pelicans on bicycles than other combinations. Catching a lab cheating specifically on my one dumb benchmark would be really funny . Dylan's methodology here - generating 1008 SVGs across an 8x6 combination - is significantly more robust than a…

What if they’re not pelicanmaxxing, but svgmaxxxing in general? Because otherwise using a LLM to generate complex svgs is pretty niche and what I thought made this a good benchmark when it was new - generalized programming and spatial knowledge. Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.

Then they’re legitimately getting better at svg which is a valuable tool.

Re: Are AI labs pelicanmaxxing?

#152

Post by usef- »

JadoJodo wrote: post #140
simonw wrote: post #17

This is fantastic I've been casually spot-checking other animals in other vehicles, because my absolute dream situation here is to catch an AI lab that's demonstrably better at pelicans on bicycles than other combinations. Catching a lab cheating specifically on my one dumb benchmark would be really funny . Dylan's methodology here - generating 1008 SVGs across an 8x6 combination - is significantly more robust than a…

It's a shame you started with pelicans on a bicycle and not snakes on a plane.

It's better to be original

Re: Are AI labs pelicanmaxxing?

#153

Post by nl »

OutOfHere wrote: post #102
qq66 wrote: post #59

Earlier quoted context omitted.

But that's a genuine worthwhile capability. It's like benchmarkmaxxing on a weightlifting competition by getting really strong.

The fear is that the SVGmaxxing is limited to "X doing Y". If such 'template maxxing' exists, it will break for other templates, e.g. "X not doing Y", "X and Y doing Z", "X doing Y doing Z", etc.

I use LLMs for 3D CAD design in OpenSCAD.

There seems to be a very strong correlation between models that are good at SVG and models that are good at 3D CAD.

Anecdote I know, but there does seem to be generalization going on here.

Re: Are AI labs pelicanmaxxing?

#154

Post by nl »

runarberg wrote: post #123
tsimionescu wrote: post #85

Earlier quoted context omitted.

No, the point is that a general-ish ability to draw good SVGs is a useful ability in itself. People need SVGs for all sorts of purposes, and if AI can generate one for them, that's mostly useful (discussions about art and employment etc notwithstanding). That said, I think this would correlate relatively little with general programming ability. They're not unrelated, of course, but being able to generate code that pa…

If people need to generate good-ish SVG why not simply use a specialized model for a much better result and for far cheaper and quicker? Why do LLMs need to be able to do this as well, but worse, slower and more expensive?

For example I do SVG diagrams to explain architecture and specific data flows in a multi-tier app.

LLMs do a great job because they understand both code and SVGs well.

Edit:

An example for a synth I'm buulding: https://imgur.com/a/U694Ek7

The irony of having to post it as a PNG isn't lost on me...

Re: Are AI labs pelicanmaxxing?

#155

Post by Dylan16807 »

netsec_burn wrote: post #50
lukev wrote: post #44

Earlier quoted context omitted.

What if they’re not pelicanmaxxing, but svgmaxxxing in general? Because otherwise using a LLM to generate complex svgs is pretty niche and what I thought made this a good benchmark when it was new - generalized programming and spatial knowledge. Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.

Addressed in the article, in case you're curious.

Addressed by three inconclusive sentences. When someone goes into deeper detail on the topic, we shouldn't assume they failed to read the article just because they said "what if" instead of mentioning those three sentences.

Re: Are AI labs pelicanmaxxing?

#156

Post by nl »

mauvehaus wrote: post #27

> All 21 pelican-bicycle images, across all seven labs, face right. No other animal/vehicle combination does that. > However, facing right is common: 60% of all 1,008 images do it. How common depends on the animal and the vehicle, and bicycles are one of the two vehicles where it’s strongest Of course the pelican on the bicycle is facing right. The drivetrain on a bicycle is on the right side. If you want any represe…

To add to to this looking at the #baaw (bikes against a wall) tag on insta It was over 200 pics before I found a pic of a bike that wasn't facing left-to-right

https://www.instagram.com/explore/search/keyword/?q=%23baaw

Re: Are AI labs pelicanmaxxing?

#157

Post by sandcat_ »

qq66 wrote: post #59
lukev wrote: post #44

Earlier quoted context omitted.

What if they’re not pelicanmaxxing, but svgmaxxxing in general? Because otherwise using a LLM to generate complex svgs is pretty niche and what I thought made this a good benchmark when it was new - generalized programming and spatial knowledge. Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.

But that's a genuine worthwhile capability. It's like benchmarkmaxxing on a weightlifting competition by getting really strong.

Yes, but the original purpose of the benchmark (simonw, please correct me if I'm wrong!) was to test whether new models were good at novel problem solving. Things they haven't been trained on. So yes, getting better at generating SVGs is great news (and it seems they have been) but this particular benchmark still strikes me as largely worthless now, unless SVGs happen to be what you care about in particular when a new model drops.

Instead, it feels like a more appropriate benchmark for the original purpose would be to come up with new, novel problems each time, and compare across all models (including previous ones).

Throwing my hat in the ring: Generate a pelican shaped crossword where all the clues are related to bicycles.

(Haiku 4.5: https://imgur.com/a/N112Nxo, I'm trying some others but it's very slow! Opus has been at it for about 20 minutes.)

Re: Are AI labs pelicanmaxxing?

#158

Post by Dylan16807 »

beering wrote: post #16
Wowfunhappy wrote: post #7

> The more plausible story is SVGmaxxing Exactly--and you have to ask yourself at this point what "maxxing" really means, since "get better at drawing SVGs" is a useful skill.

Really awful how the AI labs are skillmaxxing /s Pelicans aside, we need to remember that benchmarks are the only good quantitative way we have of comparing models. If someone has complaints about “benchmaxxing”, please ask them to contribute a better benchmark! It is valuable work and very appreciated.

Avoiding "benchmaxxing" helps keep benchmarks be a good quantitative way to compare models! It's hard to come up with new benchmarks, and it's a waste of everyone's time if we have to churn through them constantly. Effort to keep AI labs working on general skills and not "teaching to the test" is effort well spent.

Re: Are AI labs pelicanmaxxing?

#159

Post by TomatoCo »

sandcat_ wrote: post #157
qq66 wrote: post #59

Earlier quoted context omitted.

But that's a genuine worthwhile capability. It's like benchmarkmaxxing on a weightlifting competition by getting really strong.

Yes, but the original purpose of the benchmark (simonw, please correct me if I'm wrong!) was to test whether new models were good at novel problem solving. Things they haven't been trained on. So yes, getting better at generating SVGs is great news (and it seems they have been) but this particular benchmark still strikes me as largely worthless now, unless SVGs happen to be what you care about in particular when a ne…

I've mostly been running LLMs on my own hardware so the phrase "been at it for about 20 minutes" gave me pause. My first instinct was to ask "but on what hardware?" but I suppose one advantage of closed models is that there's a somewhat-consistent cognitive-effort-to-time ratio.

Re: Are AI labs pelicanmaxxing?

#160

Post by julianz »

simonw wrote: post #17

This is fantastic I've been casually spot-checking other animals in other vehicles, because my absolute dream situation here is to catch an AI lab that's demonstrably better at pelicans on bicycles than other combinations. Catching a lab cheating specifically on my one dumb benchmark would be really funny . Dylan's methodology here - generating 1008 SVGs across an 8x6 combination - is significantly more robust than a…

I asked Gemini 3.6 Flash for a pūkeko on a unicycle yesterday and the results were very stylish. The thinking notes were cool too, making sure it got all the high points of what it means to be a pūkeko (distinctive swamp hen found in NZ and Australia for anyone not familiar) and then figuring out how the legs would work with the pedals.
Post reply on HN