Gemini 2.5 Pro Preview • Hacker News Board

Re: Gemini 2.5 Pro Preview

#151

Post by sauwan » Tue, May 06, 2025, 4:51 PM UTC

jjani wrote: post #128

andy12_ wrote: post #74

Interestingly, when compering benchmarks of Experimental 03-25 [1] and Experimental 05-06 [2] it seems the new version scores slightly lower in everything except on LiveCodeBench. [1] https://storage.googleapis.com/model-cards/documents/gemini-... [2] https://deepmind.google/technologies/gemini/

Sounds like they were losing so much money on 2.5-Pro they came up with a forced update that made it cheaper to run. They can't come out with "we've made it worse across the board", nor do they want to be the first to actually raise prices, so instead they made a bit of a distill that's slightly better at coding so they can still spin it positively.

I'd be surprised if this was a new base model. It sounds like they just did some post-training RL tuning to make this version specifically stronger for coding, at the expense of other priorities.

Re: Gemini 2.5 Pro Preview

#152

Post by mhh__ » Tue, May 06, 2025, 4:52 PM UTC

killerstorm wrote: post #61

Why can't they just use version numbers instead of this "new preview" stuff? E.g. call it Gemini Pro 2.5.1.

Are you saying you find model names like o4-mini-high-pro-experimental-version5 confusing and stupid?

Re: Gemini 2.5 Pro Preview

#153

Post by simple10 » Tue, May 06, 2025, 4:52 PM UTC

jeswin wrote: post #2

Now if there was a way to add prepaid credits and monitor usage near real-time on a dashboard, like every other vendor. Hey Google are you listening?

You can do this with LLM proxies like LiteLLM. e.g. Cursor -> LiteLLM -> LLM provider API.

I have LiteLLM server running locally with Langfuse to view traces. You configure LiteLLM to connect directly to providers' APIs. This has the added benefit of being able to create LiteLLM API keys per project that proxies to different sets of provider API keys to monitor or cap billing usage.

I use https://github.com/LLemonStack/llemonstack/ to spin up local instances of LiteLLM and Langfuse.

Re: Gemini 2.5 Pro Preview

#154

Post by panarchy » Tue, May 06, 2025, 4:54 PM UTC

Is it just me that finds that while Gemini 2.5 is able to generate a lot of code that the end results are usually lackluster compared to Claude and even ChatGPT? I also find it hard-headed and frequently does things in ways I explicitly told it not to. The massive context window is pretty great though and enables me to do things I can't with the others so it still gets used a lot.

Re: Gemini 2.5 Pro Preview

#155

Post by staticman2 » Tue, May 06, 2025, 4:55 PM UTC

diggan wrote: post #118

sitkack wrote: post #85

Earlier quoted context omitted.

LLMs are extremely poor at following negative instructions, tell them what to do, not what not to do.

Ok, so saying "Implement feature X" leads to a ton of comments. How do you rewrite that comment to not include "don't write comments" while making the output not containing comments? "Write only source code, no plain text with special characters in the beginning of the line" or what are you suggesting here in practical terms?

This is sort of LLM specific. For some tasks you might try including the word comment but give the order at the beginning and end of the prompt. This is very model dependent. Like:

Refractor this. Do not write any comments.

As a reminder your task is to refractor the above code and do not write any comments.

Re: Gemini 2.5 Pro Preview

#156

Post by xyst » Tue, May 06, 2025, 4:55 PM UTC

Proprietary junk beats DeepSeek by a mere 213 points?

Oof. G and others are way behind

Re: Gemini 2.5 Pro Preview

#157

Post by scrlk » Tue, May 06, 2025, 4:56 PM UTC

panarchy wrote: post #154

Is it just me that finds that while Gemini 2.5 is able to generate a lot of code that the end results are usually lackluster compared to Claude and even ChatGPT? I also find it hard-headed and frequently does things in ways I explicitly told it not to. The massive context window is pretty great though and enables me to do things I can't with the others so it still gets used a lot.

How are you using it?

I find that I get the best results from 2.5 Pro via Google AI Studio with a low temperature (0.2-0.3).

Re: Gemini 2.5 Pro Preview

#158

Post by pzo » Tue, May 06, 2025, 4:57 PM UTC

Hawkenfall wrote: post #5

jeswin wrote: post #2

Now if there was a way to add prepaid credits and monitor usage near real-time on a dashboard, like every other vendor. Hey Google are you listening?

You can do this with https://openrouter.ai/

but if you want to use google SDK (python-genai, js-genai) rather than openai SDK (If found google api more feature rich when using different modality like audio/images/video) you cannot use openrouter. Also not sure if you are developing app and needs higher rate limits - what's typical rate limit via openrouter?

Re: Gemini 2.5 Pro Preview

#159

Post by childintime » Tue, May 06, 2025, 4:57 PM UTC

How does it perform on anything but Python and Javascript? In my experience my milage varied a lot when using C#, for example, or Zig, so I've learnt to just let it select the language it wants.

Also, why doesn't Ctrl+C work??

Re: Gemini 2.5 Pro Preview

#160

Post by benbristow » Tue, May 06, 2025, 4:57 PM UTC

ranyume wrote: post #11

I don't know if I'm doing something wrong, but every time I ask gemini 2.5 for code it outputs SO MANY comments. An exaggerated amount of comments. Sections comments, step comments, block comments, inline comments, all the gang.

You can ask it to remove the comments afterwards, and it'll do a decent job of it, but yeah, it's a pain.