Well, then you weren't thinking about the Wozniak coffee test.
> "The Coffee Test has been proposed, and is attributed to Steve Wozniak, one of the founders of Apple Computers. According to Steve, this test would require a robot, not a computer screen. The robot would need to locate the kitchen and brew a pot of coffee in a random house that it had never seen before.
And more specifically, I disagree that such a system would work just from video frames. The issue isn't so much "Coming up with text that sounds right", it's more about state and environment representation, novelty, planning multi-step processes, and so on.
A good test of if this were easy, is can it be done in game engines, which negate all the speed / repeatability / controls complexity almost entirely. Plug an LLM into a game engine with a hovering arm that can grab and place and push buttons, and have it "make coffee" in procedural generated kitchens, coffee makers, etc. Bonus points for using POV imagery, bonus points for using bags of grains that might be coffee but might not. Similar weight, feel, placement. Hell my coffee, coca, sugar, and flour are right next to each other.
That would be a landmark top tier research accomplishment and top HN article for days (the real accomplishment!)