Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My personal AI benchmark: “Generate an SVG of a frog with a Habsburg jaw”

frogs.vaguespac.es

Post reply on HN
Print view View original

51–60 of 101 posts

Re: My personal AI benchmark: “Generate an SVG of a frog with a Habsburg jaw”

#51

Post by jostylr »

krisoft wrote: post #49

Curious that none of the attempts draw the frog from side profile. If i have to draw this i would immediately know that drawing a recognisable frog is the easy part of job. Expressing a particular jaw shape and melding it on the frog is the hard part. And jaw shapes are more prominent from the side. Even absence of thinking this through you would think that some frogs will be from the front, some from the side. Just…

When I asked ChatGPT, it gave me a side profile, first as a PNG which was good and then I asked again for an SVG and it obliged, doing a version of what it produced but it was bad.

SVG: https://jostylr.com/imgs/frog_habsburg_jaw.svg

PNG: https://jostylr.com/imgs/frog_habsburg.png

Then I tried Codex with Sol 5.6 High and got a face forward one.

SVG: https://jostylr.com/imgs/frog-habsburg.svg

Re: My personal AI benchmark: “Generate an SVG of a frog with a Habsburg jaw”

#52

Post by akomtu »

dehrmann wrote: post #10

How do models approach SVG generation? In one version, I imagine them actually trying to reason about them as an LLM. In another, I imagine something closer to a GAN.

Image -> loose text description -> SVG. Did you notice that ChatGPT writes a verbose description of the image first?

Re: My personal AI benchmark: “Generate an SVG of a frog with a Habsburg jaw”

#53

Post by abound »

My personal benchmark is a directory containing a bunch of research papers on the physics of popping popcorn kernels, and a prompt about creating high fidelity, photo realistic 3D models of all the different kinds of popped kernels. Fable (surprisingly? unsurprisingly?) refused to do it last time I tried, and the results from other models are, well, fine, but there's still plenty of headroom on this particular one.

Re: My personal AI benchmark: “Generate an SVG of a frog with a Habsburg jaw”

#55

Post by acdha »

epolanski wrote: post #36

I don't get the point of these benchmarks, what are they supposed to represent practically?

The vendors like to toss around terms like “thinking” or “reasoning” to encourage prospective buyers to anthropomorphize models. These challenges are a visceral reminder that none of those marketing claims accurately describe was LLMs do: they’ll happily return things even the worst human illustrator would never hand in and make errors showing that there’s no model of the world behind anything they do.

That doesn’t mean there are no ways to use them productively but rather that you should keep in mind that the same model will happily give you code or a decision with the same level of error unless you have carefully setup a QA regimen to prevent that.

Re: My personal AI benchmark: “Generate an SVG of a frog with a Habsburg jaw”

#57

Post by uncivilized »

abound wrote: post #53

My personal benchmark is a directory containing a bunch of research papers on the physics of popping popcorn kernels, and a prompt about creating high fidelity, photo realistic 3D models of all the different kinds of popped kernels. Fable (surprisingly? unsurprisingly?) refused to do it last time I tried, and the results from other models are, well, fine , but there's still plenty of headroom on this particular one.

LLMs don’t work particularly well in 3d applications in my experience. Every new model release I’ll ask one for help with my path tracer and results are horrid.

Re: My personal AI benchmark: “Generate an SVG of a frog with a Habsburg jaw”

#59

Post by Mindless2112 »

epolanski wrote: post #36

I don't get the point of these benchmarks, what are they supposed to represent practically?

Unless the LLM has an SVG renderer in its toolbox, it's like asking a human to draw while blindfolded. It's amazing if they can do it well, but it's practically meaningless.
Post reply on HN