Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My personal AI benchmark: “Generate an SVG of a frog with a Habsburg jaw”

frogs.vaguespac.es

Post reply on HN
Print view View original

81–90 of 101 posts

Re: My personal AI benchmark: “Generate an SVG of a frog with a Habsburg jaw”

#81

Post by thebigship »

zahlman wrote: post #77
thebigship wrote: post #2

I think this one has advantages over the “pelican riding a bicycle” one because it hinges on an anatomical feature that many models associate with royalty, “habsburg” being a lineage and “habsburg jaw” being an anatomical feature. Seven of fourteen models silently imported royalty into a prompt that named only an anatomical feature. Two of them knew they were extrapolating ("because Habsburg") and did it anyway. Mist…

> Two of them knew they were extrapolating ("because Habsburg") and did it anyway. You seem to imply that they ought not to. I disagree. I wasn't familiar with the term before this post. Having learned it, were I given the task, I think I'd be strongly tempted to do the same extrapolation. > If you're deciding which model to trust with instructions, "how much does it embellish beyond what I asked" and "does it behave…

> You seem to imply that they ought not to. I disagree.

That would be like saying anyone with Lou Gehrig's disease must look like Lou Gehrig. So we'll have to agree to disagree here.

Re: My personal AI benchmark: “Generate an SVG of a frog with a Habsburg jaw”

#82

Post by Rygian »

krisoft wrote: post #49

Curious that none of the attempts draw the frog from side profile. If i have to draw this i would immediately know that drawing a recognisable frog is the easy part of job. Expressing a particular jaw shape and melding it on the frog is the hard part. And jaw shapes are more prominent from the side. Even absence of thinking this through you would think that some frogs will be from the front, some from the side. Just…

See https://news.ycombinator.com/item?id=49150393

Re: My personal AI benchmark: “Generate an SVG of a frog with a Habsburg jaw”

#83

Post by tjoff »

TonyStr wrote: post #80
DimitriBouriez wrote: post #79

Earlier quoted context omitted.

I'm afraid you'll need to look up "Habsburg" yourself.

Nathan is right though. It specifies a Habsburg jaw, not that the frog is a Habsburg king/prince. It's understandable that the LLM will hallucinate a king from this, but it isn't what's being asked of it.

It's not hallucination. The prompt lacks details and given that an AI or a human has some artistic freedom. After all, that is why current AI is usable at all.

Re: My personal AI benchmark: “Generate an SVG of a frog with a Habsburg jaw”

#87

Post by acdha »

zahlman wrote: post #76
acdha wrote: post #55

Earlier quoted context omitted.

The vendors like to toss around terms like “thinking” or “reasoning” to encourage prospective buyers to anthropomorphize models. These challenges are a visceral reminder that none of those marketing claims accurately describe was LLMs do: they’ll happily return things even the worst human illustrator would never hand in and make errors showing that there’s no model of the world behind anything they do. That doesn’t m…

> That doesn’t mean there are no ways to use them productively but rather that you should keep in mind that the same model will happily give you code or a decision with the same level of error unless you have carefully setup a QA regimen to prevent that. The standard retort seems to be "this is also true of a large proportion of humans". But I think it's clear that there are differing patterns in how humans vs. model…

Exactly right: it’s not that humans were perfect-I’ve seen too many write-offs from the big consulting companies to ever say that—but that the pattern of errors is different and any intuition you have around that will not be correct in the LLM era.

Re: My personal AI benchmark: “Generate an SVG of a frog with a Habsburg jaw”

#88

Post by inigyou »

jnwatson wrote: post #56

Fable 5 on Max knocks it out of the park: https://imgur.com/a/usR8K7G Definitely has some creative flourishes. (I made no extra prompting. Just the above text. Single shot.)

Uh, it failed the test by assuming it was royalty.
Post reply on HN