Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

blog.jcz.dev

Post reply on HN
Print view View original

41–50 of 97 posts

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#41

Post by ctoth »

andrepd wrote: post #8
wild_pointer wrote: post #3

I wonder how much of it is due to the model being familiar with the game or parts of it, be it due to training of the game itself, or reading/watching walkthroughs online.

There was a well-publicised "Claude plays Pokémon" stream where Claude failed to complete Pokemon Blue in spectacular fashion, despite weeks of trying. I think only a very gullible person would assume that future LLMs didn't specifically bake this into their training, as they do for popular benchmarks or for penguins riding a bike.

> as they do for popular benchmarks or for penguins riding a bike.

Citation?

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#42

Post by nomel »

re-thc wrote: post #25
mkoubaa wrote: post #21

Earlier quoted context omitted.

I can't believe how massively underpaid I was when I was 11

Do you hallucinate as a kid?

Kids definitely do this. They fill in blanks/context with assumptions, resulting in all sorts of silly responses, for topics of sparse knowledge/certainty. They're not lying, because they think it's true. Sometimes the gap filling is wrong, but usually downright brilliant, within the context of their knowledge.

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#44

Post by giancarlostoro »

soulofmischief wrote: post #11
giancarlostoro wrote: post #10

Earlier quoted context omitted.

I have to think they need to know enough of the guides for the game for it to work out, how do they know whats on screen?

In my project I rigged up an in-browser emulator and directly fed captured images of the screen to local multimodal models. So it just looks right at what's going on, writes a description for refinement, and uses all of that to create and manage goals, write to a scratchpad and submit input. It's minimal scaffolding because I wanted to see what these raw models are capable of. Kind of a benchmark.

I have a feeling if you gave them access to GameFAQ guides they might be able to play better, but it depends on how you can feed them the data.

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#45

Post by murukesh_s »

sussmannbaka wrote: post #43

So after years of being gleefully told that AI will replace all jobs an omniscient state of the art model, with heavy assistance, takes more than two weeks and thousands of dollars in tokens to do what child me did in a few days? Huh.

I used to think the same until latest agents started adding perfectly fine features to a large existing react app with just basic input (in English) . Most of the jobs require levels of intelligence below that. It's just a matter of time before agents get to that.

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#46

Post by kqr »

zurfer wrote: post #30
squimmy26 wrote: post #15

How certain can we be that these improvements aren't just a result of Gemini 3 Pro pre-training on endless internet writeups of where 2.5 has struggled (and almost certainly what a human would have done instead)? In other words, how much of this improvement is true generalization vs memorization?

You're too kind. Even the CEO of Google retweeted how well Gemini 2.5 did on Pokemon. There is a high chance that now it's explicitly part of the training regime. We kind of need a different kind of game to know how well it generalizes.

I have a draft doing this with text adventures: https://entropicthoughts.com/updated-llm-benchmark

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#48

Post by blauditore »

murukesh_s wrote: post #45
sussmannbaka wrote: post #43

So after years of being gleefully told that AI will replace all jobs an omniscient state of the art model, with heavy assistance, takes more than two weeks and thousands of dollars in tokens to do what child me did in a few days? Huh.

I used to think the same until latest agents started adding perfectly fine features to a large existing react app with just basic input (in English) . Most of the jobs require levels of intelligence below that. It's just a matter of time before agents get to that.

It's about the complexity of the task. Front end apps tend do be much less complex and boilerplate-y than backends, hence AI tends to work better.

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#49

Post by elif »

oceansky wrote: post #4

"Crucially, it tells the agent not to rely on its internal training data (which might be hallucinated or refer to a different version of the game) but to ground its knowledge in what it observes. " Does this even have any effect?

I would imagine that prompting anything like this will have an excessively ironic effect like convincing it to suppress patterns which it would consider to be pre-knowledge.

If you looked inside they would be spinning on something like "oh I know this is the tile to walk on, but I have to only rely on what I observe! I will do another task instead to satisfy my conditions and not reveal that I have pre-knowledge.

LLMs are literal douche genies. The less you say, generally, the better

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#50

Post by etse »

blauditore wrote: post #48
murukesh_s wrote: post #45

Earlier quoted context omitted.

I used to think the same until latest agents started adding perfectly fine features to a large existing react app with just basic input (in English) . Most of the jobs require levels of intelligence below that. It's just a matter of time before agents get to that.

It's about the complexity of the task. Front end apps tend do be much less complex and boilerplate-y than backends, hence AI tends to work better.

Isn’t frontend more complex? If my task starts with a Figma UI design, how well does a code agent do at generating working code that looks right, and iterate on it (presuming some browser MCP)? Some automated tests seem enough for an genetic loop on backend.
Post reply on HN