Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

blog.jcz.dev

Post reply on HN
Print view View original

91–97 of 97 posts

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#91

Post by MrCheeze »

jwrallie wrote: post #2

Being through the game recently, I am not surprised Goldenrod Underground was a challenge, it is very confusing and even though I solved it through trial and error, I still don't know what I did. Olivine Lighthouse is the real surprise, as it felt quite obvious to me.

This writeup on the underground puzzle is worth reading, it's a pretty baffling "puzzle" design. https://pokemow.com/Gen2/ShutterPuzzle/

That said, it's definitely Gem's fault that it struggled so long, considering it ignored the NPCs that give clues.

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#92

Post by bjackman »

elephanlemon wrote: post #23
bbondo wrote: post #12

1.88 billion tokens * $12 / 1M tokens (output) suggests a total cost of $22,560 to solve the game with Gemini 3 Pro?

“Gemini 3 Pro was often overloaded, which produced long spans of downtime that 2.5 Pro experienced much less often” I was unclear if this meant that the API was overloaded or if he was on a subscription plan and had hit his limit for the moment. Although I think that the Gemini plans just use weekly limits, so I guess it must be API.

Geminii CLI has a specific "model is overloaded" error message which is distinct from "you're out of quota" so I suspect whatever tools they're using for this probably have something similar, and they're referring to that.

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#93

Post by blauditore »

murukesh_s wrote: post #51
blauditore wrote: post #48

Earlier quoted context omitted.

It's about the complexity of the task. Front end apps tend do be much less complex and boilerplate-y than backends, hence AI tends to work better.

I disagree - having worked on backends most of the time, I find modern frontend much more complex (and difficult to test) than pure backend. When I say modern frontend - its mostly React, state management like Redux, Zustand, Router framework like React Router, a CSS framework like Tailwind and component framework like Shadcn. Not to mention different versions of React, different ways of managing state, animation/tra…

Actual complexity depends is not imposed by the framework, but by the application itself, and the amount of front-end logic tends to be lower. Yes, there is more complexity in the setup of front end code (now there are dependencies and build pipelines), but ultimately they should simplofy things especially for slighly more complex websites.

Testing is one of the things that's generally tedious in front end applications, but not inherently complex. There may be lots of config needed (e.g. for setting up and controlling a headless browser), and long turnarounds because tests are slow and shaky. But they are also boilerplatey.

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#95

Post by giancarlostoro »

soulofmischief wrote: post #64
giancarlostoro wrote: post #44

Earlier quoted context omitted.

I have a feeling if you gave them access to GameFAQ guides they might be able to play better, but it depends on how you can feed them the data.

It turns out that cutting edge super small (3b param etc) models that fit in the browser are not great at playing Pokémon on an even basic level, even navigation is difficult when only providing raw visual information, and object recognition of the low-resolution sprites is not great. So I lost interest before even getting to the point of providing specific strategy. But, it runs in browser and works with any supplie…

You should consider publishing your setup as either a blog post or a GitHub repo. I think it could make for good benchmarking of smaller models. Ideally we can one day all run small models that can do amazing things.

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#96

Post by murukesh_s »

rafaelmn wrote: post #61
murukesh_s wrote: post #51

Earlier quoted context omitted.

I disagree - having worked on backends most of the time, I find modern frontend much more complex (and difficult to test) than pure backend. When I say modern frontend - its mostly React, state management like Redux, Zustand, Router framework like React Router, a CSS framework like Tailwind and component framework like Shadcn. Not to mention different versions of React, different ways of managing state, animation/tra…

That's just a farmiliarity thing. I've worked on project doing full web FE, mobile and BE. It's hard to generalize but modern frontend is very good at isolating you from dealing with complex state machine states and you're dealing with single user/limited concurrency. It's usually easy to find all references/usecases for something. Most modern backend is building consistent distributed state machines, you need to cov…

>Most modern backend is building consistent distributed state machines, you need to cover all the edge cases, deal with concurrency, different clients/contracts etc. I would say getting BE right (beyond simple CRUD) is going to be hard for LLM simply because the context is usually wider and hard to compress/isolate.

Seeing the kind of complexity that agents (not standalone llm) are able to navigate - I can only start to believe - just a matter of time it can do all kinds of programming, including state of the art backend programming - even writing a database on its own - good thing with backend is its easily testable and if there is documentation that a developer can read and comprehend - an llm/agent would be able to do that - not very far from today.

Re: Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

#97

Post by nomel »

johnebgd wrote: post #71
nomel wrote: post #42

Earlier quoted context omitted.

Kids definitely do this. They fill in blanks/context with assumptions, resulting in all sorts of silly responses, for topics of sparse knowledge/certainty. They're not lying, because they think it's true. Sometimes the gap filling is wrong, but usually downright brilliant, within the context of their knowledge.

Are you sure there is an age limit for that kind of behavior in humans?

I replied within the context provided rather than all possible contexts. Would you also like to bring up the interpolation and extrapolation seen during problem solving in the cuddle fish, since it's related?
Post reply on HN