171–180 of 253 posts
Earlier quoted context omitted.
But that's a genuine worthwhile capability. It's like benchmarkmaxxing on a weightlifting competition by getting really strong.
https://www.youtube.com/watch?v=jgYYOUC10aM reminds me of this Key and Peele skit
Earlier quoted context omitted.
But that's a genuine worthwhile capability. It's like benchmarkmaxxing on a weightlifting competition by getting really strong.
Yes, but the original purpose of the benchmark (simonw, please correct me if I'm wrong!) was to test whether new models were good at novel problem solving. Things they haven't been trained on. So yes, getting better at generating SVGs is great news (and it seems they have been) but this particular benchmark still strikes me as largely worthless now, unless SVGs happen to be what you care about in particular when a ne…
This is fantastic I've been casually spot-checking other animals in other vehicles, because my absolute dream situation here is to catch an AI lab that's demonstrably better at pelicans on bicycles than other combinations. Catching a lab cheating specifically on my one dumb benchmark would be really funny . Dylan's methodology here - generating 1008 SVGs across an 8x6 combination - is significantly more robust than a…
What if they’re not pelicanmaxxing, but svgmaxxxing in general? Because otherwise using a LLM to generate complex svgs is pretty niche and what I thought made this a good benchmark when it was new - generalized programming and spatial knowledge. Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.
This is fantastic I've been casually spot-checking other animals in other vehicles, because my absolute dream situation here is to catch an AI lab that's demonstrably better at pelicans on bicycles than other combinations. Catching a lab cheating specifically on my one dumb benchmark would be really funny . Dylan's methodology here - generating 1008 SVGs across an 8x6 combination - is significantly more robust than a…
And that's how we end up with PelicanSkynet
i'm puzzled by the choice to have an llm judge the images.
Even if using LLM is the only way to do things at scale it does not mean that it's always the right tool.
Same pelican grid + MacBook Pro in 3D. Also with end cost.