Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

blog.jcz.dev

Post reply on HN
Print view View original

61–70 of 97 posts

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#61

Post by rafaelmn »

murukesh_s wrote: post #51
blauditore wrote: post #48

Earlier quoted context omitted.

It's about the complexity of the task. Front end apps tend do be much less complex and boilerplate-y than backends, hence AI tends to work better.

I disagree - having worked on backends most of the time, I find modern frontend much more complex (and difficult to test) than pure backend. When I say modern frontend - its mostly React, state management like Redux, Zustand, Router framework like React Router, a CSS framework like Tailwind and component framework like Shadcn. Not to mention different versions of React, different ways of managing state, animation/tra…

That's just a farmiliarity thing. I've worked on project doing full web FE, mobile and BE.

It's hard to generalize but modern frontend is very good at isolating you from dealing with complex state machine states and you're dealing with single user/limited concurrency. It's usually easy to find all references/usecases for something.

Most modern backend is building consistent distributed state machines, you need to cover all the edge cases, deal with concurrency, different clients/contracts etc. I would say getting BE right (beyond simple CRUD) is going to be hard for LLM simply because the context is usually wider and hard to compress/isolate.

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#62

Post by dwaltrip »

sussmannbaka wrote: post #43

So after years of being gleefully told that AI will replace all jobs an omniscient state of the art model, with heavy assistance, takes more than two weeks and thousands of dollars in tokens to do what child me did in a few days? Huh.

Children are incredibly smart. All of this was fantasy 15 years ago. Comments like yours are amazing to me…

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#63

Post by dwaltrip »

andrepd wrote: post #8
wild_pointer wrote: post #3

I wonder how much of it is due to the model being familiar with the game or parts of it, be it due to training of the game itself, or reading/watching walkthroughs online.

There was a well-publicised "Claude plays Pokémon" stream where Claude failed to complete Pokemon Blue in spectacular fashion, despite weeks of trying. I think only a very gullible person would assume that future LLMs didn't specifically bake this into their training, as they do for popular benchmarks or for penguins riding a bike.

If they game the pelican benchmark, it’d be pretty obvious.

Just try other random, non-realistic things like “a giraffe walking a tightrope”, “a car sitting at a cafe eating a pizza”, etc.

If the results are dramatically different, then they gamed it. If they are similar in quality, then they probably didn’t.

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#64

Post by soulofmischief »

giancarlostoro wrote: post #44
soulofmischief wrote: post #11

Earlier quoted context omitted.

In my project I rigged up an in-browser emulator and directly fed captured images of the screen to local multimodal models. So it just looks right at what's going on, writes a description for refinement, and uses all of that to create and manage goals, write to a scratchpad and submit input. It's minimal scaffolding because I wanted to see what these raw models are capable of. Kind of a benchmark.

I have a feeling if you gave them access to GameFAQ guides they might be able to play better, but it depends on how you can feed them the data.

It turns out that cutting edge super small (3b param etc) models that fit in the browser are not great at playing Pokémon on an even basic level, even navigation is difficult when only providing raw visual information, and object recognition of the low-resolution sprites is not great. So I lost interest before even getting to the point of providing specific strategy.

But, it runs in browser and works with any supplied ROM, none of it is Pokémon-specific so I should set aside time to serve it and make the code available

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#66

Post by mchusma »

rybosome wrote: post #57
sussmannbaka wrote: post #43

So after years of being gleefully told that AI will replace all jobs an omniscient state of the art model, with heavy assistance, takes more than two weeks and thousands of dollars in tokens to do what child me did in a few days? Huh.

“And, because AI never got any better or any cheaper after that point, sussmanbaka’s wry observation remained true in perpetuity, forever.” - History, most likely

Cost per intelligence is shrinking by something like 100x per year. Even the Gemini flash release would potentially do as well for 1/5th already.

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#68

Post by wavemode »

jdiff wrote: post #19
hypron wrote: post #14

Earlier quoted context omitted.

My issue with this is that the LLM could just be roleplaying that it doesn't know.

Of course it is. It's not capable of actually forgetting or suppressing its training data. It's just double checking rather than assuming because of the prompt. Roleplaying is exactly what it's doing. At any point, it may stop doing that and spit out an answer solely based on training data. It's a big part of why search overview summaries are so awful. Many times the answers are not grounded in the material.

It may actually have the opposite effect - the instruction to not use prior knowledge may have been what caused Gemini 3 to assume incorrect details about how certain puzzles worked and get itself stuck for hours. It knew the right answer (from some game walkthrough in its training data), but intentionally went in a different direction in order to pretend that it didn't know. So, paradoxically, the results of the test end up worse than if the model truly didn't know.

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#70

Post by jug »

orbital-decay wrote: post #56

The baked-in assumptions observation is basically the opposite of the impression I get after watching Gemini 3's CoT. With the maximum reasoning effort it's able to break out of the wrong route by rethinking the strategy. For example I gave it an onion address without the .onion part, and told it to figure out what this string means. All reasoning models including Gemini 2.5 and 3 assume it's a puzzle or a cipher (be…

I've had some weird "thinking outside the box" behavior like this. I once asked 3 Pro what Ozzy Osbourne is up to. The CoT was a journey, I can tell you! It's not in its training data that he actually passed away. It did know he was planning a tour though. It had a real struggle trying to consolidate "suspicious search results" and even questioned whether it was fake news, or running against a simulation!, determining it wasn't going to fall for my "test".

It did ultimately decide Ozzy was alive. I pushed back on that, and it instantly corrected itself and partially blamed my query "what is he up to" for being formulated as if he was alive.

Post reply on HN