But they're not recreating Minecraft. They're recreating one of the most common tutorials on the internet that shows a simple voxel overworld and nothing else. That's not Minecraft.
Recreating Minecraft Is Not a Benchmark
41–50 of 81 posts
Re: Recreating Minecraft Is Not a Benchmark
#42[deleted]
Re: Recreating Minecraft Is Not a Benchmark
#43But they're not recreating Minecraft. They're recreating one of the most common tutorials on the internet that shows a simple voxel overworld and nothing else. That's not Minecraft.
Important distinction, but I would debate the "most common tutorials on the internet" claim.
Re: Recreating Minecraft Is Not a Benchmark
#44Very low quality content. Unsurprisingly pangram 100%.
I'm not sure myself, there are a lot of comma splices and other such grammar mistakes. Maybe just edited with AI?
Re: Recreating Minecraft Is Not a Benchmark
#45I used to think that way about SVGBench, after all labs can just train on the test set, right? It turns out the task was highly generalizable. Try designing a logo and you quickly see the gap between models visually. Even though there is still a gap between "shiny demo SVG" and actual real-world use.
Same thing happened with MineBench, basically SVGBench+3D, until that got "saturated".
Remember spinning hexagon bench[1]? Or the AI World Clocks[2]? Yeah, that used to be hard for frontier models.
Creating games is the next iteration that still has some signal left. Assets + Game logic + UI + Sound, it let's you assess a model's "taste" very quickly.
What else is left, once all these benchmarks get saturated?
Re: Recreating Minecraft Is Not a Benchmark
#46I don't remember seeing playstation controller SVGs before, I doubt they benchmaxxed that
Re: Recreating Minecraft Is Not a Benchmark
#47Definitely not.
It has to be Doom or Crysis, aren't they the ones people usually ask if it can run?
Re: Recreating Minecraft Is Not a Benchmark
#48I use Minecraft (and Warcraft and an arcade flying simulator) as a silly but directionally correct indication of the models' capabilitites. Compare Astra[0] with GPT 5.4[1] which was OpenAI's state of the art just six months ago. (all tests on more models with code and prompts available here: https://senko.net/vibecode-bench ) Yes, it's not a scientific benchmark but it's a good heuristic. For a better eval, create a…
I really like these tests for what its worth and I see them on youtube sometimes. I would like to ask a few things though
TLDR: Basically focusing on recreating pay to win (mobile or otherwise) games and recreating them non pay to win perhaps instead of focusing on recreation of minecraft for benchmarks could have a genuinely meaningful impact, and making these games portable as well could be another interesting idea. [so it can be played on any operating system/device so using web or if native then for (Android/IOS/Linux/Windows/MacOS) using game engines like (Preferably godot)/Unity/UE.]
Could the test focus more on pay to win games with unique dynamics.
For example: I literally wanted to create a clash royale recreation because clash royale is a highly pay to win game.
The game is unique enough to have memories but is pay to win enough that it ragebaits me as to what its current situation is, its so pay to win now. A recreation would have genuine effect whereas yet another minecraft clone wouldn't.
I have some fond memories of the game and my brother and I used to play it (my brother moreso than me). Also clash of clans and clash royale famously prevented windows users. So I remember downloading bluestacks to play it on laptop but it required 2GB of ram and back then we only had 1GB. (Ironic that we might come back to that time)
Another question that I have for you which I have been genuinely curious is: who is footing the bill for these benchmarks and youtube videos. What are the economics surrounding it?
I imagine the bill to run quite hot sometimes and I find running these benchmarks to be quite unaffordable personally.
I also wish to ask if you have any theories as to why not people on Youtube share their videos. I found this [0] Minecraft clone by Fable 5.1 extremely good yet they haven't shared the source. I am unsure as to what exact reasons might be behind most youtube videos on recreation with AI to not share the actual code. I don't find much rationale in not sharing AI generated code of a recreation of a game especially if one is making a video about it. So thanks for once for actually sharing the output code as well as I surprisingly found it to be a bit rare!
Re: Recreating Minecraft Is Not a Benchmark
#49Earlier quoted context omitted.
> ridiculously polished detail Can you point me to one of these? The only Astra game I tried was the incredibly janky Mario Kart clone from OpenAI where you could fly by spamming jump
I was impressed by this 3D version of Zork. Video: https://bsky.app/profile/emollick.bsky.social/post/3muqcwa4j... Demo: https://zork-underground-empire.netlify.app Code: https://github.com/emollick/zork-underground-empire Prompts in a screenshot here: https://bsky.app/profile/emollick.bsky.social/post/3muqdyefc... - I extracted them as text here: https://gist.github.com/simonw/dab1843ef17edeaeffd18b0acbba2...
Impressed like that's cool ai is doing it? Or just impressed in general like its high quality?
Re: Recreating Minecraft Is Not a Benchmark
#50I keep seeing Astra make beautiful 3d stuff online, yet when I feed it some old school RuneScape assets (even tried with some very detailed guidelines) and asked it to generate some new plausible assets it failed horribly. I think there’s still something really off with current (frontier) models when it comes to creating “novel” stuff? Even 2004 style graphics.. Or am promoting it wrong?
It's a veeeeery specific artstyle