Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

blog.jcz.dev

Post reply on HN
Print view View original

11–20 of 97 posts

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#11

Post by soulofmischief »

giancarlostoro wrote: post #10
soulofmischief wrote: post #5

Nice writeup! I need to start blogging about my antics. I rigged up several cutting edge small local models to an emulator all in-browser and unsuccessfully tried to get them to play different Pokémon games. They just weren't as sharp as the frontier models. This was a good while back but I'm sure a lot of people might find the process and code interesting even if it didn't succeed. Might resurrect that project.

I have to think they need to know enough of the guides for the game for it to work out, how do they know whats on screen?

In my project I rigged up an in-browser emulator and directly fed captured images of the screen to local multimodal models.

So it just looks right at what's going on, writes a description for refinement, and uses all of that to create and manage goals, write to a scratchpad and submit input. It's minimal scaffolding because I wanted to see what these raw models are capable of. Kind of a benchmark.

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#13

Post by criley2 »

andrepd wrote: post #8
wild_pointer wrote: post #3

I wonder how much of it is due to the model being familiar with the game or parts of it, be it due to training of the game itself, or reading/watching walkthroughs online.

There was a well-publicised "Claude plays Pokémon" stream where Claude failed to complete Pokemon Blue in spectacular fashion, despite weeks of trying. I think only a very gullible person would assume that future LLMs didn't specifically bake this into their training, as they do for popular benchmarks or for penguins riding a bike.

While it is true that model makers are increasingly trying to game benchmarks, it's also true that benchmark-chasing is lowering model quality. GPT 5, 5.1 and 5.2 have been nearly universally panned by almost every class of user, despite being a benchmark monster. In fact, the more OpenAI tries to benchmark-max, the worse their models seem to get.

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#14

Post by hypron »

ragibson wrote: post #9
oceansky wrote: post #4

"Crucially, it tells the agent not to rely on its internal training data (which might be hallucinated or refer to a different version of the game) but to ground its knowledge in what it observes. " Does this even have any effect?

Yes, at least to some extent. The author mentions that the base model knows the answer to the switch puzzle but does not execute it properly here. "It is worth noting that the instruction to "ignore internal knowledge" played a role here. In cases like the shutters puzzle, the model did seem to suppress its training data. I verified this by chatting with the model separately on AI Studio; when asked directly multiple…

My issue with this is that the LLM could just be roleplaying that it doesn't know.

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#15

Post by squimmy26 »

How certain can we be that these improvements aren't just a result of Gemini 3 Pro pre-training on endless internet writeups of where 2.5 has struggled (and almost certainly what a human would have done instead)?

In other words, how much of this improvement is true generalization vs memorization?

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#17

Post by brianwawok »

hypron wrote: post #14
ragibson wrote: post #9

Earlier quoted context omitted.

Yes, at least to some extent. The author mentions that the base model knows the answer to the switch puzzle but does not execute it properly here. "It is worth noting that the instruction to "ignore internal knowledge" played a role here. In cases like the shutters puzzle, the model did seem to suppress its training data. I verified this by chatting with the model separately on AI Studio; when asked directly multiple…

My issue with this is that the LLM could just be roleplaying that it doesn't know.

To test would just need to edit the rom and switch around the solution. Not sure how complicated that is, likely depends on the rom system.

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#18

Post by Workaccount2 »

tootyskooty wrote: post #7
oceansky wrote: post #4

"Crucially, it tells the agent not to rely on its internal training data (which might be hallucinated or refer to a different version of the game) but to ground its knowledge in what it observes. " Does this even have any effect?

I'm wondering about this too. Would be nice to see an ablation here, or at least see some analysis on the reasoning traces. It definitely doesn't wipe its internal knowledge of Crystal clean (that's not how LLMs work). My guess is that it slightly encourages the model to explore more and second-guess it's likely very-strong Crystal game knowledge but that's about it.

The model probably recognizes the need for a grassroots effort to solve the problem, to "show it's work".

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#19

Post by jdiff »

hypron wrote: post #14
ragibson wrote: post #9

Earlier quoted context omitted.

Yes, at least to some extent. The author mentions that the base model knows the answer to the switch puzzle but does not execute it properly here. "It is worth noting that the instruction to "ignore internal knowledge" played a role here. In cases like the shutters puzzle, the model did seem to suppress its training data. I verified this by chatting with the model separately on AI Studio; when asked directly multiple…

My issue with this is that the LLM could just be roleplaying that it doesn't know.

Of course it is. It's not capable of actually forgetting or suppressing its training data. It's just double checking rather than assuming because of the prompt. Roleplaying is exactly what it's doing. At any point, it may stop doing that and spit out an answer solely based on training data.

It's a big part of why search overview summaries are so awful. Many times the answers are not grounded in the material.

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#20

Post by baby »

oceansky wrote: post #4

"Crucially, it tells the agent not to rely on its internal training data (which might be hallucinated or refer to a different version of the game) but to ground its knowledge in what it observes. " Does this even have any effect?

Do we have examples of this in promps in other contexts?
Post reply on HN