Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are AI labs pelicanmaxxing?

dylancastillo.co

Post reply on HN
Print view View original

81–90 of 253 posts

Re: Are AI labs pelicanmaxxing?

#81

Post by HarHarVeryFunny »

stri8ted wrote: post #37

You seem to assume training on pelican would not result in improved performance on other similar tasks. Why?

Either you've memorized the outline (or detailed component shapes) of, say, a horse, or you haven't. Memorizing the outline of a pelican isn't going to help you with the horse.

You could train a model to do something a bit different like a pencil sketch, or vector graphic sketch, of something given a photo of it, and expect that to be a generalized skill, but if you are asking the model to do it "from memory" then memorizing a pelican is no substitute for not having memorized a horse.

Re: Are AI labs pelicanmaxxing?

#82

Post by docheinestages »

zahlman wrote: post #74
docheinestages wrote: post #62

Earlier quoted context omitted.

I think a more fundamental test is SVG art creation in general. Perhaps a pipeline to take any image, caption it, ask the LLM for an SVG, rasterize to an image, and finally either use a deterministic visual similarity check or ask another LLM to be the judge and score how close the SVG is to the original image.

Fidelity to the original is definitely not how humans would measure "art" in this context.

True, maybe we can call the generated SVG something else than art.

Re: Are AI labs pelicanmaxxing?

#83

Post by cherioo »

dllu wrote: post #24

I feel like getting LLMs to spit out an SVG is akin to getting a human artist to draw something by just reciting a list of coordinates. It's insanely hard and unnatural. Image generation models nowadays can easily generate a photorealistic pelican riding a bicycle, where the bicycle has perfect structure. But it is, of course, only a raster image. It seems that we're missing a kind of step to decompose an image into…

I don’t quite agree. Good human artist can visualize in their mind how to draw a picture, i think. Which i think is no different than LLM doing SVG drawing in their “head”. Anthropic’s recent post call this head-space “workspace”.

It just might feel foreign to human who does not have a SVG trained head-space.

Re: Are AI labs pelicanmaxxing?

#84

Post by Balgair »

qq66 wrote: post #59
lukev wrote: post #44

Earlier quoted context omitted.

What if they’re not pelicanmaxxing, but svgmaxxxing in general? Because otherwise using a LLM to generate complex svgs is pretty niche and what I thought made this a good benchmark when it was new - generalized programming and spatial knowledge. Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.

But that's a genuine worthwhile capability. It's like benchmarkmaxxing on a weightlifting competition by getting really strong.

https://www.youtube.com/watch?v=jgYYOUC10aM

reminds me of this Key and Peele skit

Re: Are AI labs pelicanmaxxing?

#85

Post by tsimionescu »

sysguest wrote: post #66
qq66 wrote: post #59

Earlier quoted context omitted.

But that's a genuine worthwhile capability. It's like benchmarkmaxxing on a weightlifting competition by getting really strong.

well that holds IF svgmaxxing is 100% "code-writing-maxxing" ...which.. hmm I dunno if they are same or not

No, the point is that a general-ish ability to draw good SVGs is a useful ability in itself. People need SVGs for all sorts of purposes, and if AI can generate one for them, that's mostly useful (discussions about art and employment etc notwithstanding).

That said, I think this would correlate relatively little with general programming ability. They're not unrelated, of course, but being able to generate code that paints an accurate + esthetically pleasing image is quite different from generating code that achieves a non-spatial goal.

Re: Are AI labs pelicanmaxxing?

#86

Post by apwheele »

robocat wrote: post #61
apwheele wrote: post #23

So this is not my experience at all for asking about simple SVG icons for web-pages. Here is one of the examples I have tried for in the past, make a simple cartoon SVG knife for a map icon for a crime map. https://x.com/CrimeDecoder/status/2080008114615537766 Can see the images for ChatGPT/Claude (Sonnet 5), and Gemini are all quite bad. Jagged edge of LLMs. How do you explain being able to generate very complicated…

Does asking for a dagger help?

If you look at the raster image ChatGPT generated, that is fine. It is just this example (and other simple SVG icons I have asked for) result in pretty bad SVGs. It just makes me highly suspicious that the LLMs are learning shape primitives and extrapolating to new shapes, vs just having a big dictionary of prior examples and stitching them together.

Re: Are AI labs pelicanmaxxing?

#87

Post by bnfcl »

wasabi991011 wrote: post #72
bnfcl wrote: post #39

This is funny, I actually did a similar experiment just yesterday. Looking for evidence of the same, but with another twist: checking if the models would choose to create a pelican on a bicycle, if no specific bird or method of transportation was specified. My version of it: https://www.modelbias.ai/pelican-on-a-bicycle-test

I find your analysis much more convincing than TFA, since it doesn't require a subjective evaluation and is more robust to animal/transport complexity.

Thanks! Because I think that models are becoming better at creating SVGs in general. If you look at Claude Fable 5 and Kimi K3 for example.

In my tests it did create bicycles the most, but this is just a general bias I believe, as tested here: https://www.modelbias.ai/prompt/transport

Re: Are AI labs pelicanmaxxing?

#88

Post by zahlman »

bnfcl wrote: post #39

This is funny, I actually did a similar experiment just yesterday. Looking for evidence of the same, but with another twist: checking if the models would choose to create a pelican on a bicycle, if no specific bird or method of transportation was specified. My version of it: https://www.modelbias.ai/pelican-on-a-bicycle-test

Simple as they are, there are some really aesthetically pleasing penguins on skateboards in there, including from less capable models. (In fact, I would say the Opus series got progressively worse at it over time.)

Re: Are AI labs pelicanmaxxing?

#90

Post by kaliqt »

lukev wrote: post #44
simonw wrote: post #17

This is fantastic I've been casually spot-checking other animals in other vehicles, because my absolute dream situation here is to catch an AI lab that's demonstrably better at pelicans on bicycles than other combinations. Catching a lab cheating specifically on my one dumb benchmark would be really funny . Dylan's methodology here - generating 1008 SVGs across an 8x6 combination - is significantly more robust than a…

What if they’re not pelicanmaxxing, but svgmaxxxing in general? Because otherwise using a LLM to generate complex svgs is pretty niche and what I thought made this a good benchmark when it was new - generalized programming and spatial knowledge. Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.

Funnily enough, not that niche, because I have tried many times to do it as part of a wider project.
Post reply on HN