Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Recreating Minecraft Is Not a Benchmark

kuber.studio

Post reply on HN
Print view View original

51–60 of 81 posts

Re: Recreating Minecraft Is Not a Benchmark

#51

Post by Imustaskforhelp »

RomanKornev wrote: post #45

I used to think that way about SVGBench, after all labs can just train on the test set, right? It turns out the task was highly generalizable. Try designing a logo and you quickly see the gap between models visually. Even though there is still a gap between "shiny demo SVG" and actual real-world use. Same thing happened with MineBench, basically SVGBench+3D, until that got "saturated". Remember spinning hexagon bench…

> What else is left, once all these benchmarks get saturated?

I was thinking this thing quite recently and wish to write a blog post about it but IMO, I believe that the next AI benchmarks would be profitable businessmaking. There are/were already some trials done (The famous WSJ/Anthropic one[0]) and some shop iirc in SF which is doing this and recent example of trying to start a lemonade stand with AI[1]

In my opinion, it can't really get more meta than that plus on the more humour side, it could help these companies make some dollars as they desperately need it.

I always wonder if these AI models can self-autonomously generate profit, why would these AI companies try to let you generate the main profit while you just pay them one time or a very small amount in tokens. Why should these AI giants allow a person to be a middle man in the first place in some sense and instead not just run these agents autonomously.

Maybe it could be because these AI agents might not be the best in such a meta task (money making/business) but as such, it does feel to me that it might be a good benchmark in the first place.

[0]: We Let AI Run a Vending Machine. It Lost All the Money. | WSJ: https://www.youtube.com/watch?v=SpPhm7S9vsQ

[1]: Everything AI Does When You Ask It to Start a Lemonade Stand: https://www.youtube.com/watch?v=6Ide5pRLR8Y

Re: Recreating Minecraft Is Not a Benchmark

#52

Post by bluegatty »

Right answer wrong thesis.

The example of 'labs are optimizing for this' is the wrong thesis.

AI arbitrarily generating something of 'apparent sophistication' is not that hard - being able to produce it to spec that has invariable vague elements - and then being able to rationally modify it is the problem.

Look at image gen: you press the magic button and get a 'Pelican on a Bike' - but you can't just change the 'hat' of the Pelican. You have to press the magic button again, and you get a whole different Pelican on a Bike.

This is the fundamental conceit.

It's akin to the conceit that 'writing the code is the work' - but it's not - it's the research, the design, docs, integration and all that 'know-how' that has to 'sit somewhere' so the shape of the thing output can be adapted and moulded.

Arbitrary code output is not quite worthless but almost, it's the the '80% that means another 80% and then anther 80% to go'. It's like a nicer stating point.

The research capabilities of the AI, which don't make for nice demos, are arguably more powerful.

Re: Recreating Minecraft Is Not a Benchmark

#53

Post by simonw »

tayo42 wrote: post #49
simonw wrote: post #33

Earlier quoted context omitted.

I was impressed by this 3D version of Zork. Video: https://bsky.app/profile/emollick.bsky.social/post/3muqcwa4j... Demo: https://zork-underground-empire.netlify.app Code: https://github.com/emollick/zork-underground-empire Prompts in a screenshot here: https://bsky.app/profile/emollick.bsky.social/post/3muqdyefc... - I extracted them as text here: https://gist.github.com/simonw/dab1843ef17edeaeffd18b0acbba2...

Impressed like that's cool ai is doing it? Or just impressed in general like its high quality?

I think it's a genuinely good implementation of the concept. It looks good and plays reasonably well and captures what it needs to capture from the original.

(Update: OK the sword fighting isn't good. As I got deeper into the game my positive first impressions wore off.)

Re: Recreating Minecraft Is Not a Benchmark

#54

Post by Imustaskforhelp »

arjie wrote: post #40

Very low quality content. Unsurprisingly pangram 100%.

On some occasions, as I have come to understand Hackernews, its not really about the content but sometimes just about the Title or literally just the sentiment behind it.

Maybe we are all being shown recreation of Minecraft or other video games as a benchmark and maybe there is a discussion to be had of the statement of (is it a benchmark or not). I, for one, just appreciate that we are all here who could talk about this topic that I was just thinking about and can have a good discussion about.

In some sense, its speaking some thought that everyone (many) might be thinking internally out loud.

Also on a related topic, I have a question that I wish to ask, there are many text files that I have which are me just writing extremely crude thoughts. I once tried to let AI re-write it in a blog format and I found it to somewhat express what I was saying, perhaps saving me hours.

I still don't upload those blog posts because I don't wish to write AI generated blog posts even though I might have spent more than hour writing those original thoughts. To me, its because I don't exactly know how to disclose things. Some of the original thoughts for example might be quite PII/sensitive. Should I share the exact prompts or the original thoughts and what are the correct hygienic ettiquetes towards this nuanced topic?

It would also take me more than hours to edit the posts and I might still not be confident in my ability of trying to create that blog post and might end up never finishing that public blog post in the first place due to procastination.

I also don't just want to sometimes write AI generated blog posts because its a slippery slope. Without proper disclosure, the other person might not even know if another human on another side of planet even gave a thought about it or not. For all intents they might think that nobody really wrote it but only an AI and just "write a blog post about X"

I also would dislike being the person on that end and I also dislike reading AI generated posts. It produces visceral reaction of hatred and closing the tab when I read AI generated slop.

I ironically wanted to create a blog post about this nuance but I couldn't capture it and I then had AI write that blog post but I didn't post it. I have had some very nuanced meta experiences about this exact topic to be honest and I would love to hear some thoughts & opinions about it.

I have the thoughts but because of all of this, in some sense, a lack of means to share. I would wish to hear other peoples thoughts about those thoughts that I had and if other people resonate with it or not but alas, some of them are just text files within my computer.

Re: Recreating Minecraft Is Not a Benchmark

#55

Post by letmevoteplease »

BalinKing wrote: post #44
arjie wrote: post #40

Very low quality content. Unsurprisingly pangram 100%.

I'm not sure myself, there are a lot of comma splices and other such grammar mistakes. Maybe just edited with AI?

It reads like it was generated with AI and then edited to have poorer grammar.

"A launch is a first impression and first impressions are marketing, that’s why you’re always bound to be shocked - the shock was scheduled."

"And if a model scores well but keeps failing at your work, that is actually a gap that deserves investigation."

That's Claude speaking.

Re: Recreating Minecraft Is Not a Benchmark

#56

Post by RomanKornev »

Imustaskforhelp wrote: post #51
RomanKornev wrote: post #45

I used to think that way about SVGBench, after all labs can just train on the test set, right? It turns out the task was highly generalizable. Try designing a logo and you quickly see the gap between models visually. Even though there is still a gap between "shiny demo SVG" and actual real-world use. Same thing happened with MineBench, basically SVGBench+3D, until that got "saturated". Remember spinning hexagon bench…

> What else is left, once all these benchmarks get saturated? I was thinking this thing quite recently and wish to write a blog post about it but IMO, I believe that the next AI benchmarks would be profitable businessmaking. There are/were already some trials done (The famous WSJ/Anthropic one[0]) and some shop iirc in SF which is doing this and recent example of trying to start a lemonade stand with AI[1] In my opin…

I would be surprised if we're not there yet. If Jane Street can generate $500 million per MW of compute right now I'm sure it's already possible.

We might still be figuring out how to benchmark these models by the time next gen comes.

Re: Recreating Minecraft Is Not a Benchmark

#57

Post by BlobberSnobber »

lukan wrote: post #43
jere wrote: post #41

But they're not recreating Minecraft. They're recreating one of the most common tutorials on the internet that shows a simple voxel overworld and nothing else. That's not Minecraft.

Important distinction, but I would debate the "most common tutorials on the internet" claim.

I think you meant “pedantic distinction”. Everyone reading that comment understands that recreating Minecraft is a very common tutorial for game development, not literally one of the most common tutorials on the internet in general.

Re: Recreating Minecraft Is Not a Benchmark

#58

Post by tripleee »

simonw wrote: post #33
tripleee wrote: post #9

Earlier quoted context omitted.

> ridiculously polished detail Can you point me to one of these? The only Astra game I tried was the incredibly janky Mario Kart clone from OpenAI where you could fly by spamming jump

I was impressed by this 3D version of Zork. Video: https://bsky.app/profile/emollick.bsky.social/post/3muqcwa4j... Demo: https://zork-underground-empire.netlify.app Code: https://github.com/emollick/zork-underground-empire Prompts in a screenshot here: https://bsky.app/profile/emollick.bsky.social/post/3muqdyefc... - I extracted them as text here: https://gist.github.com/simonw/dab1843ef17edeaeffd18b0acbba2...

The scale of this is impressive. I think if you showed me a single puzzle / dungeon though it's something we've seen on par from previous models.

First impressions of the code quality also feel like it needs some serious refactoring, but then again I'm not a game dev:

https://github.com/emollick/zork-underground-empire/blob/cod...

https://github.com/emollick/zork-underground-empire/blob/cod...

Overall impression though yeah it's crazy AI can do this.

Re: Recreating Minecraft Is Not a Benchmark

#59

Post by Imustaskforhelp »

bluegatty wrote: post #52

Right answer wrong thesis. The example of 'labs are optimizing for this' is the wrong thesis. AI arbitrarily generating something of 'apparent sophistication' is not that hard - being able to produce it to spec that has invariable vague elements - and then being able to rationally modify it is the problem. Look at image gen: you press the magic button and get a 'Pelican on a Bike' - but you can't just change the 'hat…

> Look at image gen: you press the magic button and get a 'Pelican on a Bike' - but you can't just change the 'hat' of the Pelican. You have to press the magic button again, and you get a whole different Pelican on a Bike.

I do understand what you mean and perhaps I am trying to treat it as a problem to be solved and challenge accepted but my first thoughts are if its an vector image like SVG (the famous simonw's pelican on a bike benchmark)

Then you could in theory have a layered approach and then just change the code of the hat so for example (IIRC) Code.

So basically I am suggesting modularizing of concerns of areas so that you could have better autonomy over what exact thing you wish to change.

Though I imagine that you and I might be saying the same thing and you are suggesting that the crux of the argument is exactly that you need specific know how in reaching to that said modularity where you can best use AI.

When you might need the generation of the SVG as compared to image generation because you realize that your project might need the pelican changing lots of hats and image gen wouldn't be feasible but for that to actually say to AI, You might need to know in the first place that SVG or (HTML?) might be better use cases for this. Again, I must admit that I am not an expert in this so I can't absolutely comment on what the best thing for this particular use case could be and maybe that might be your point.

Have we arrived at similar conclusion or perhaps, is there more nuance to it?

On a side note: I actually once had a real use case of where I needed layered approach similar to Figma but generated through AI. preferably something which can just work through good ol chat app UI which could give an index.html or other code for this modification/layered approach.

Like recently there is https://bento.page which has been somewhat similar for this in some sense but for pdf's. So is there something different but for image-ish thing? If there is an expert lurking here, I would love to know the answer and gain some knowledge about it, thanks :-D

Re: Recreating Minecraft Is Not a Benchmark

#60

Post by cheevly »

bluegatty wrote: post #52

Right answer wrong thesis. The example of 'labs are optimizing for this' is the wrong thesis. AI arbitrarily generating something of 'apparent sophistication' is not that hard - being able to produce it to spec that has invariable vague elements - and then being able to rationally modify it is the problem. Look at image gen: you press the magic button and get a 'Pelican on a Bike' - but you can't just change the 'hat…

[flagged]
Post reply on HN