71–80 of 101 posts
Fable 5 on Max knocks it out of the park: https://imgur.com/a/usR8K7G Definitely has some creative flourishes. (I made no extra prompting. Just the above text. Single shot.)
rex paludis is a nice touch.
My personal benchmark is a directory containing a bunch of research papers on the physics of popping popcorn kernels, and a prompt about creating high fidelity, photo realistic 3D models of all the different kinds of popped kernels. Fable (surprisingly? unsurprisingly?) refused to do it last time I tried, and the results from other models are, well, fine , but there's still plenty of headroom on this particular one.
LLMs don’t work particularly well in 3d applications in my experience. Every new model release I’ll ask one for help with my path tracer and results are horrid.
Fable 5 on Max knocks it out of the park: https://imgur.com/a/usR8K7G Definitely has some creative flourishes. (I made no extra prompting. Just the above text. Single shot.)
I don't get the point of these benchmarks, what are they supposed to represent practically?
The vendors like to toss around terms like “thinking” or “reasoning” to encourage prospective buyers to anthropomorphize models. These challenges are a visceral reminder that none of those marketing claims accurately describe was LLMs do: they’ll happily return things even the worst human illustrator would never hand in and make errors showing that there’s no model of the world behind anything they do. That doesn’t m…
The standard retort seems to be "this is also true of a large proportion of humans". But I think it's clear that there are differing patterns in how humans vs. models err on various tasks.
I think this one has advantages over the “pelican riding a bicycle” one because it hinges on an anatomical feature that many models associate with royalty, “habsburg” being a lineage and “habsburg jaw” being an anatomical feature. Seven of fourteen models silently imported royalty into a prompt that named only an anatomical feature. Two of them knew they were extrapolating ("because Habsburg") and did it anyway. Mist…
You seem to imply that they ought not to. I disagree.
I wasn't familiar with the term before this post. Having learned it, were I given the task, I think I'd be strongly tempted to do the same extrapolation.
> If you're deciding which model to trust with instructions, "how much does it embellish beyond what I asked" and "does it behave deterministically" are directly practical questions.
Agency is agency. You still need to vet what the model's output is actually permitted to control.
Check out my MacBook SVG benchmark. From my experience, it demonstrates the Real model’s behavior. However, I notice the errors it makes, which are similar to the mistakes made by the mistake model in code. https://playcode.io/blog/macbook-svg-benchmark
Fable 5 on Max knocks it out of the park: https://imgur.com/a/usR8K7G Definitely has some creative flourishes. (I made no extra prompting. Just the above text. Single shot.)
It makes the frog a king which is not in the prompt, so there is a bit of confusion.
Earlier quoted context omitted.
It makes the frog a king which is not in the prompt, so there is a bit of confusion.
I'm afraid you'll need to look up "Habsburg" yourself.