I don't get the point of these benchmarks, what are they supposed to represent practically?
The ability of an LLM to produce something not in its training data set.
[deleted]
My personal AI benchmark: “Generate an SVG of a frog with a Habsburg jaw”
41–50 of 101 posts
Re: My personal AI benchmark: “Generate an SVG of a frog with a Habsburg jaw”
#42My personal human benchmark: "Jump on one leg, while reciting the national anthem of Latvia, translated to Spanish, backwards, while drawing a frog with a brush held by toes of the other leg, on the ceiling". So far they're not doing very good but I'm sure they'll improve over time.
[dead]
Re: My personal AI benchmark: “Generate an SVG of a frog with a Habsburg jaw”
#43No gpt-5.6 sol and no fable?
Re: My personal AI benchmark: “Generate an SVG of a frog with a Habsburg jaw”
#44Here is GLM 5.2 (https://codeinput.com/s/HAO0qTxw2ia) which is still inferior to Opus. I can't find Qwen 3.8 which now is my daily driver replacing GLM. This SVG test matches my experience when working with the different models. The other models can get the details right but their output is structured in a way that makes little or no sense.
I also did a timeline from 4.7 to 5.2: https://codeinput.com/s/7oK2IIA7qRO The improvements in models looks much less impressive with this test.
Re: My personal AI benchmark: “Generate an SVG of a frog with a Habsburg jaw”
#45The secret to great interview questions and challenge tests is keeping them secret. Posting them on HN and getting them onto the front page puts them in jeopardy.
Re: My personal AI benchmark: “Generate an SVG of a frog with a Habsburg jaw”
#46I don't get the point of these benchmarks, what are they supposed to represent practically?
For me this looks ideological (or even political), not practical. The theory is that LLMs are approaching general intelligence (whatever that means) and that the more generic of a task they can perform—no matter how badly—the closer we are to AGI. Specialized models can do this a lot better and for far cheaper then LLMs, but because people are so politically invested in a single statistical model being able to outper…
[deleted]
Re: My personal AI benchmark: “Generate an SVG of a frog with a Habsburg jaw”
#47[flagged]
Re: My personal AI benchmark: “Generate an SVG of a frog with a Habsburg jaw”
#48Am I the only one who thinks it's incredible that an LLM can do this, and at the same time it's ridiculous to expect it to be capable of doing it, even thought it clearly can do it?
Re: My personal AI benchmark: “Generate an SVG of a frog with a Habsburg jaw”
#49Curious that none of the attempts draw the frog from side profile. If i have to draw this i would immediately know that drawing a recognisable frog is the easy part of job. Expressing a particular jaw shape and melding it on the frog is the hard part. And jaw shapes are more prominent from the side.
Even absence of thinking this through you would think that some frogs will be from the front, some from the side. Just by chance. And yet all appears to go for the harder pose.
Re: My personal AI benchmark: “Generate an SVG of a frog with a Habsburg jaw”
#50How do models approach SVG generation? In one version, I imagine them actually trying to reason about them as an LLM. In another, I imagine something closer to a GAN.
I would be interested to hear more about that too. SVGs seem to be one of the biggest challenges- it has to write reasoned code rather than simply find averages of rasterized pixels. One thing I noticed is that none of the models chose to draw the frog in profile which would have made the jaw shape more prominent. To me that suggests the reasoning is very limited: "draw a frog" and "add feature X". The statistically average frog in the SVG training data is apparently front-facing. I tried the prompt in ChatGPT images (not SVG) and it produced a photo-realistic image of the frog in profile, showing the jaw clearly. Then when I asked it to convert the image to a cartoon vector it fell back to a generic front portrait template similar to those shown in the benchmark, nothing like the profile it had just produced.