Recreating Minecraft Is Not a Benchmark

A scripting API makes the problem much more approachable, but what about those videos where Astra is drawing people from a photo? Here's one using canva: https://x.com/iam_zachi/status/2095992132620136677

Is that also using scripting to batch updates? It does look as if the mouse is moving.

Could be hallucination, but I gave this to an LLM and this is what it suggested:

"

The workflow shown in the video—processing an image and then controlling a computer interface to draw it—is a combination of two well-established fields: Computer Vision and UI Automation.

You do not necessarily need a Large Language Model to perform the underlying image processing; standard algorithms can do this deterministically.

Step A: Image Processing (The "Brain")

You can write a script (using Python libraries like OpenCV or Pillow) to process the reference photo:

    - Edge Detection: Use filters (like Canny or Sobel) to find the "high spatial frequencies" (outlines).

    - Color Quantization: Use clustering algorithms (like K-Means) in the HSL space to group millions of pixels into a small palette of distinct colors.

    - Vectorization: Convert these processed shapes into a set of coordinates (SVG paths) that represent exactly where the mouse needs to move.

Once the image is converted into a set of instructions (coordinates and color codes), you can use automation tools to physically control the computer and draw on Canva.

    - Browser Automation: Developers have already created projects that use Selenium (a web automation tool) combined with edge detection algorithms to draw images onto HTML canvases.

    - The script reads the pixel data, calculates the mouse coordinates, and executes the "click-and-drag" actions in the browser.

    - Computer Use APIs: In the case of GPT-6 Astra, the model uses a "Computer Use" interface. It effectively takes the processed image data (or generates it internally) and outputs high-level commands (e.g., "Move mouse to X,Y," "Click," "Select Hex Color #FF5733"), which the system then executes on the screen.

Seems plausible and easier to believe. Also, feels like a "magic trick" designed to fool the user into believing that the agent is drawing interactively by using its vision, since it could just have written a python script that takes the input image, and produces the exact same result without automating the screen.

As to why the prompts aren't shared, for these more complex things it's most likely a somewhat messy process (ie. not a single-prompt one-shot creation; some back & forth) that would make the whole thing seem less spectacular.

Likewise for the end result - it's probably cherry-picked what works well. For example, in Claude models' resuts I always get stuck in water (something about height/jump calculations is off), where with Astra I didn't have that problem. These sorts of issues you can only spot if you try to playthrough yourself.

This is just my speculation tho. As for me, I do the tests because I'm interested in the results (easy comparison across time & models) - then I started sharing them because people asked.

I have OpenAI and Anthropic subscriptions so testing these is not an extra cost for me (I'm sloppy around recording the tokens & API-equivalent cost tho - have to improve on this). For the other models, it's total a few bucks per month or so.

So if you're careful about the cost, it's not too much, especially for a serious youtuber who's doing it for commercial reasons.

Finally regarding your comment about cloning the popular enshittified games - I don't think people are going to be doing that for testing, but if you want to have a different spin (or do a close-enough clone for yourself) on a game you loved, the modern AI systems can often deliver!

[0] from the link you posted "i am in disbelief, this is insane, bro what is this, you can't believe it's ai" - yeah...umm, it's not that good :)

Either way, you'd have to do more work to claim that providers are specifically optimizing for Minecraft vs. other hypotheses like Minecraft just having more resources, preexisting implementations, and base model knowledge than other games.

If we're just going by vibes, I'd wager we're at the point that it's more straightforward to improve the model generally than try to special-case specific demos that are popular on social media.