Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are AI labs pelicanmaxxing?

dylancastillo.co

Post reply on HN
Print view View original

211–220 of 253 posts

Re: Are AI labs pelicanmaxxing?

#211

Post by phreeza »

sandcat_ wrote: post #157
qq66 wrote: post #59

Earlier quoted context omitted.

But that's a genuine worthwhile capability. It's like benchmarkmaxxing on a weightlifting competition by getting really strong.

Yes, but the original purpose of the benchmark (simonw, please correct me if I'm wrong!) was to test whether new models were good at novel problem solving. Things they haven't been trained on. So yes, getting better at generating SVGs is great news (and it seems they have been) but this particular benchmark still strikes me as largely worthless now, unless SVGs happen to be what you care about in particular when a ne…

I think a lot of LLM progress in the last 2 years or so is really from densifying the training data to niches like this and not so much from general problem solving capability. I guess the hope is that the deluge of super specific training data will pay off through better overall competence. cf https://www.dwarkesh.com/p/the-sample-efficiency-black-hole

Re: Are AI labs pelicanmaxxing?

#213

Post by 21asdffdsa12 »

somenameforme wrote: post #201
nl wrote: post #197

Earlier quoted context omitted.

ChatGPT 5.5 and Sol 5.6, Fable are good. I haven't really tested Opus 4.8, but 4.7 wasn't nearly as good as ChatGPT 5.5.

Interesting! The reason I asked is I've had poor results with Fable and 3d stuff. Its spatial awareness seems poor - doing things like rotating left and then right back, and then pitching nonsensically, just to try to capture a segment of a scene for a verification pass. And its placement often results in clipping, misrotations, and so on. It could well be that the exact domain matters more than the bigger picture co…

The failure seems to be that a scene is inherently hierarchical with a graph with offset, rotation, scale and a name - and this seems to be transferred back poorely fro example during 3d generation.

Re: Are AI labs pelicanmaxxing?

#214

Post by tempfile »

qq66 wrote: post #59
lukev wrote: post #44

Earlier quoted context omitted.

What if they’re not pelicanmaxxing, but svgmaxxxing in general? Because otherwise using a LLM to generate complex svgs is pretty niche and what I thought made this a good benchmark when it was new - generalized programming and spatial knowledge. Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.

But that's a genuine worthwhile capability. It's like benchmarkmaxxing on a weightlifting competition by getting really strong.

Ugh, they're just goodmaxxing.

Re: Are AI labs pelicanmaxxing?

#215

Post by 21asdffdsa12 »

timClicks wrote: post #114
lukev wrote: post #44

Earlier quoted context omitted.

What if they’re not pelicanmaxxing, but svgmaxxxing in general? Because otherwise using a LLM to generate complex svgs is pretty niche and what I thought made this a good benchmark when it was new - generalized programming and spatial knowledge. Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.

If they're optimizing for SVG generation, then that's an excellent outcome in my opinion. Vector images shouldn't be "pretty niche".

And SVG is fantastic to draw easter eggs or hidden messages. Just a insult to the top of the 4 in 404.

Re: Are AI labs pelicanmaxxing?

#217

Post by OliverGuy »

sheept wrote: post #167
cheesecakegood wrote: post #138

Earlier quoted context omitted.

Interestingly, there was an artist a few years back who (for an unrelated project) had almost 400 people across a range of ages draw a bicycle and 75% of those faced left! So this seems to actually go slightly against the human drawing intuition. https://www.gianlucagimini.it/portfolio-item/velocipedia/ On the other hand, I notice that the prompt says to draw a pelican riding a bicycle, implying motion... and since m…

That's interesting. I think the bikes-facing-left bias probably comes from how humans use bikes: the kickstand is on the left, so people likely hold and approach bikes from its left. Looking online, the kickstand is apparently on the left to avoid the gears. Based on the other comment about bike photography, it's interesting the same design choice makes humans and cameras/LLMs see bikes from different sides.

If you are a bike nerd and like taking photos of your bike(s) you'll almost always try and have it drive train towards the camera, which would be front on the right side. Possibly this is where is this coming from? Generally also true for retail listings too

Re: Are AI labs pelicanmaxxing?

#218

Post by rldjbpin »

the pelican bike combo is truly an HN phenomenon, and i think the hypothesis carry a lot of weight about the assumption of the mindshare this website truly has in the industry.

very nice approach to test it and might be a nice way to "grid search" evals in other use cases perhaps.

Re: Are AI labs pelicanmaxxing?

#220

Post by zkmon »

simonw wrote: post #17

This is fantastic I've been casually spot-checking other animals in other vehicles, because my absolute dream situation here is to catch an AI lab that's demonstrably better at pelicans on bicycles than other combinations. Catching a lab cheating specifically on my one dumb benchmark would be really funny . Dylan's methodology here - generating 1008 SVGs across an 8x6 combination - is significantly more robust than a…

Why is it hard to come up with tests that closely resemble real-world usecases where the models are being used?
Post reply on HN