Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini-2.5-pro-preview-06-05

deepmind.google

Post reply on HN
Print view View original

121–130 of 237 posts

Re: Gemini-2.5-pro-preview-06-05

#121

Post by xtracto »

wiradikusuma wrote: post #56

I have two issues with Gemini that I don't experience with Claude: 1. It RENAMES VARIABLE NAMES even in places where I don't tell it to change (I pass them just as context). and 2. Sometimes it's missing closing square brackets. Sure I'm a lazy bum, I call the variable "json" instead of "jsonStringForX", but it's contextual (within a closure or function), and I appreciate the feedback, but it makes reviewing the chan…

I have a very clear example of Gemini getting it wrong:

For a code like this, it keeps changing processing_class=tokenizer to "tokenizer=tokenizer", even though the parameter was renamed and even after adding the all caps comment.

    #Set up the SFTTrainer
    print("Setting up SFTTrainer...")
    trainer = SFTTrainer(
    model=model,
    train_dataset=train_dataset,
    args=sft_config,
    processing_class=tokenizer, # DO NOT CHANGE. THIS IS NOW THE CORRECT PROPERTY NAME
    )
    print("SFTTrainer ready.")
I haven't tried with this latest version, but the 05-06 pro still did it wrong.

Re: Gemini-2.5-pro-preview-06-05

#122

Post by johnfn »

Alifatisk wrote: post #116
johnfn wrote: post #12

Impressive seeing Google notch up another ~25 ELO on lmarena, on top of the previous #1, which was also Gemini! That being said, I'm starting to doubt the leaderboards as an accurate representation of model ability. While I do think Gemini is a good model, having used both Gemini and Claude Opus 4 extensively in the last couple of weeks I think Opus is in another league entirely. I've been dealing with a number of gn…

> after a bit Gemini would spin in circles or actually (I've never seen this before!) give up and say it can't do it No way, is there any way to see the dialog or recreate this scenario!?

The chat was in Cursor, so I don't know a way to provide a public link, but here is the last paragraph that it output before I (and it) gave up. I honestly could have re-prompted it from scratch and maybe it would have gotten it, but at this point I was pretty sure that even if it did, it was going to make a total mess of things. Note that it was iterating on a test failure and had spun through multiple attempts at this point:

> Given the persistence of the error despite multiple attempts to refine the type definitions, I'm unable to fix this specific TypeScript error without a more profound change to the type structure or potentially a workaround that might compromise type safety or accuracy elsewhere. The current type definitions are already quite complex.

The two prior paragraphs, in case you're curious:

> I suspect the issue might be a fundamental limitation or bug in how TypeScript is resolving these highly recursive and conditional types when they are deeply nested. The type system might be "giving up" or defaulting to a less specific type ({ __raw: T }) prematurely.

> Since the runtime logic seems to be correctly hydrating the nested objects (as the builder.build method recursively calls hydrateHelper), the problem is confined to the type system's ability to represent this.

I found, as you can see in the first of the prior two paragraphs, that Gemini often wanted to claim that the issue was on TypeScript's side for some of these more complex issues. As proven by Opus, this simply wasn't the case.

Re: Gemini-2.5-pro-preview-06-05

#123

Post by johnfn »

varunneal wrote: post #39
johnfn wrote: post #12

Impressive seeing Google notch up another ~25 ELO on lmarena, on top of the previous #1, which was also Gemini! That being said, I'm starting to doubt the leaderboards as an accurate representation of model ability. While I do think Gemini is a good model, having used both Gemini and Claude Opus 4 extensively in the last couple of weeks I think Opus is in another league entirely. I've been dealing with a number of gn…

Have you tried o3 on those problems? I've found o3 to be much more impressive than Opus 4 for all of my use cases.

To be honest, I haven't, because the "This model is extremely expensive" popup on Cursor makes me a bit anxious - but given the accolades here I'll have to give it a shot.

Re: Gemini-2.5-pro-preview-06-05

#124

Post by baw-bag »

svachalek wrote: post #53
throwaway314155 wrote: post #46

Earlier quoted context omitted.

It's interesting you say that because o3, while being a considerable improvement over OpenAI's other models, still doesn't match the performance of Opus 4 and Gemini 2.5 Pro by a long shot for me. However, o3 resides in the ChatGPT app, which is still superior to the other chat apps in many ways, particularly the internet search implementation works very well.

If you're coding through chat apps you're really behind the times. Try an agent IDE or plugin.

I am really struggling with this. I tried Cline with both OpenAI and Claude to very weird results. Often burning through credits to get no where or just running out of context. I just got Cursor for a try so can't say anything on that yet.

Re: Gemini-2.5-pro-preview-06-05

#125

Post by Tostino »

tymonPartyLate wrote: post #120
johnfn wrote: post #12

Impressive seeing Google notch up another ~25 ELO on lmarena, on top of the previous #1, which was also Gemini! That being said, I'm starting to doubt the leaderboards as an accurate representation of model ability. While I do think Gemini is a good model, having used both Gemini and Claude Opus 4 extensively in the last couple of weeks I think Opus is in another league entirely. I've been dealing with a number of gn…

I just realized that Opus 4 is the first model that produced "beautiful" code for me. Code that is simple, easy to read, not polluted with comments, no unnecessary crap, just pretty, clean and functional. I had my first "wow" moment with it in a while. That being said it occasionally does something absolutely stupid. Like completely dumb. And when I ask it "why did you do this stupid thing", it replies "oh yeah, you'…

My issue is that every time i've attempted to use Opus 4 to solve any problem, I would burn through my usage cap within a few min and not have solved the problem yet because it misunderstood things about the context and I didn't get the prompt quite right yet.

With Sonnet, at least I don't run out of usage before I actually get it to understand my problem scope.

Re: Gemini-2.5-pro-preview-06-05

#126

Post by johnfn »

zamadatix wrote: post #109
johnfn wrote: post #12

Impressive seeing Google notch up another ~25 ELO on lmarena, on top of the previous #1, which was also Gemini! That being said, I'm starting to doubt the leaderboards as an accurate representation of model ability. While I do think Gemini is a good model, having used both Gemini and Claude Opus 4 extensively in the last couple of weeks I think Opus is in another league entirely. I've been dealing with a number of gn…

I think the only way to be particularly impressed with new leading models lately is to hold the opinion all of the benchmarks are inaccurate and/or irrelevant and it's vibes/anecdotes where the model is really light years ahead. Otherwise you look at the numbers on e.g. lmarena and see it's claiming a ~16% preference win rate for gpt-3.5-turbo from November of 2023 over this new world-leading model from Google.

Not sure I follow - Gemini has ELO 1470, GPT3.5-turbo is 1206, which is an 86% win rate. https://chatgpt.com/share/6841f69d-b2ec-800c-9f8c-3e802ebbc0...

Re: Gemini-2.5-pro-preview-06-05

#127

Post by joshmlewis »

baw-bag wrote: post #124
svachalek wrote: post #53

Earlier quoted context omitted.

If you're coding through chat apps you're really behind the times. Try an agent IDE or plugin.

I am really struggling with this. I tried Cline with both OpenAI and Claude to very weird results. Often burning through credits to get no where or just running out of context. I just got Cursor for a try so can't say anything on that yet.

It's a skill that takes some persistence and trial and error. Happy to chat with you about it if you want to send me an email.

Re: Gemini-2.5-pro-preview-06-05

#128

Post by baw-bag »

joshmlewis wrote: post #127
baw-bag wrote: post #124

Earlier quoted context omitted.

I am really struggling with this. I tried Cline with both OpenAI and Claude to very weird results. Often burning through credits to get no where or just running out of context. I just got Cursor for a try so can't say anything on that yet.

It's a skill that takes some persistence and trial and error. Happy to chat with you about it if you want to send me an email.

I really appreciate that. I will see how I get on and may well give you a shout. Thank you!

Re: Gemini-2.5-pro-preview-06-05

#129

Post by throwaway314155 »

svachalek wrote: post #53
throwaway314155 wrote: post #46

Earlier quoted context omitted.

It's interesting you say that because o3, while being a considerable improvement over OpenAI's other models, still doesn't match the performance of Opus 4 and Gemini 2.5 Pro by a long shot for me. However, o3 resides in the ChatGPT app, which is still superior to the other chat apps in many ways, particularly the internet search implementation works very well.

If you're coding through chat apps you're really behind the times. Try an agent IDE or plugin.

I think this is debatable. But I've used Cursor and various extensions for VS Code. They're all fine (but cursor can fuck all the way off for stealing the `code` shell integration from VS Code) but you don't _need_ an IDE as Claude Code has shown us (currently my primary method of vibe coding).

It's mostly about the cost though. Things are far more affordable in the the various apps/subscriptions. Token-priced API's can get very expensive very quickly.

Re: Gemini-2.5-pro-preview-06-05

#130

Post by macNchz »

danielbln wrote: post #97
wiradikusuma wrote: post #56

I have two issues with Gemini that I don't experience with Claude: 1. It RENAMES VARIABLE NAMES even in places where I don't tell it to change (I pass them just as context). and 2. Sometimes it's missing closing square brackets. Sure I'm a lazy bum, I call the variable "json" instead of "jsonStringForX", but it's contextual (within a closure or function), and I appreciate the feedback, but it makes reviewing the chan…

Gemini loves to add idiotic non-functional inline comments. "# Added this function" "# Changed this to fix the issue" No, I know, I was there! This is what commit messages for, not comments that are only relevant in one PR.

I love when I ask it to remove things and it doesn't want to truly let go, so it leaves a comment instead:

   # Removed iterMod variable here because it is no longer needed.
It's like it spent too much time hanging out with an engineer who doesn't trust version control and prefers to just comment everything out.

Still enjoying Gemini 2.5 Pro more than Claude Sonnet these days, though, purely on vibes.

Post reply on HN