Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 2.5 Pro Preview

developers.googleblog.com

Post reply on HN
Print view View original

221–230 of 728 posts

Re: Gemini 2.5 Pro Preview

#221

Post by breppp »

Benjammer wrote: post #25
ranyume wrote: post #11

I don't know if I'm doing something wrong, but every time I ask gemini 2.5 for code it outputs SO MANY comments. An exaggerated amount of comments. Sections comments, step comments, block comments, inline comments, all the gang.

I've found that heavily commented code can be better for the LLM to read later, so it pulls in explanatory comments into context at the same time as reading code, similar to pulling in @docs, so maybe it's doing that on purpose?

I always thought these were there to ground the LLM on the task and produce better code, an artifact of the fact that this will autocomplete better based on past tokens. Similarly always thought this is why ChatGPT always starts every reply with repeating exactly what you asked again

Re: Gemini 2.5 Pro Preview

#222

Post by laborcontract »

okdood64 wrote: post #203
laborcontract wrote: post #79

My guess is that they've done a lot of tuning to improve diff based code editing. Gemini 2.5 is fantastic at agentic work, but it still is pretty rough around the edges in terms of generating perfectly matching diffs to edit code. It's probably one of the very few issues with the model. Luckily, aider tracks this. They measure the old gemini 2.5 generating proper diffs 92% of the time. I bet this goes up to ~95-98% h…

What do you mean by agentic work in this context?

Knowing when to call functions, generating the proper function calling text structure, properly executing functions in sequence, knowing when it's completed its objective, and doing that over an extended context window.

Re: Gemini 2.5 Pro Preview

#223

Post by arnaudsm »

jjani wrote: post #129
arnaudsm wrote: post #126

Earlier quoted context omitted.

They should have shown the benchmarks. Or market it as a coding model, like Qwen & Mistral.

That's clearly not a PR angle they could possibly take when it's replacing the overall SotA model. This is a business decision, potentially inference cost related.

From a business pov it's a great move, for the customers it's evil to hide evidence that your product became worse.

Re: Gemini 2.5 Pro Preview

#224

Post by tough »

blensor wrote: post #50
GaggiX wrote: post #15

Earlier quoted context omitted.

You can ask to not use comments or use less comments, you can put this in the system prompt too.

Maybe too many comments could be a good metric to check if someone just yolo accepted the result or if they actually checked if it's correct. I don't have problems with getting lot's of comments in the output, I am just deleting it while reading what it did

another great tell of code reviewers yolo'ing it is that LLM's usually put the full filename path on the output, so if you see a file with the filename / path on the first line, thats prob a llm output

Re: Gemini 2.5 Pro Preview

#225

Post by tasuki »

ctxc wrote: post #182
pizza wrote: post #174

Earlier quoted context omitted.

Are you sure about that? Try these.. - (1e(1e10) + 1) - 1e(1e10) - sqrt(sqrt(2)) * sqrt(sqrt(2)) * sqrt(sqrt(2)) * sqrt(sqrt(2))

Three decades and I haven't had to do anything remotely resembling this on a calculator, much less find the calculator wrong. Same for the majority of general population I assume.

The person you're replying to pointed out that you shouldn't expect a calculator to be 100% accurate 100% of the time. Especially not when faced with adversarial prompts.

Re: Gemini 2.5 Pro Preview

#226

Post by [deleted] »

jjani wrote: post #128
andy12_ wrote: post #74

Interestingly, when compering benchmarks of Experimental 03-25 [1] and Experimental 05-06 [2] it seems the new version scores slightly lower in everything except on LiveCodeBench. [1] https://storage.googleapis.com/model-cards/documents/gemini-... [2] https://deepmind.google/technologies/gemini/

Sounds like they were losing so much money on 2.5-Pro they came up with a forced update that made it cheaper to run. They can't come out with "we've made it worse across the board", nor do they want to be the first to actually raise prices, so instead they made a bit of a distill that's slightly better at coding so they can still spin it positively.

[deleted]

Re: Gemini 2.5 Pro Preview

#227

Post by johnisgood »

segphault wrote: post #18

My frustration with using these models for programming in the past has largely been around their tendency to hallucinate APIs that simply don't exist. The Gemini 2.5 models, both pro and flash, seem significantly less susceptible to this than any other model I've tried. There are still significant limitations, no amount of prompting will get current models to approach abstraction and architecture the way a person doe…

> hallucinate APIs

Tell me about it. Thankfully I have not experienced it as much with Claude as I did with GPT. It can get quite annoying. GPT kept telling me to use this and that and none of them were real projects.

Re: Gemini 2.5 Pro Preview

#228

Post by danenania »

lukeschlather wrote: post #108
killerstorm wrote: post #61

Why can't they just use version numbers instead of this "new preview" stuff? E.g. call it Gemini Pro 2.5.1.

I take preview to mean the model may be retired on an accelerated timescale and replaced with a "real" model so it's dangerous to put into prod unless you are paying attention.

Scheduled tasks in ChatGPT are useful for keeping track of these kinds of things. You can have it check daily whether there's a change in status, price, etc. for a particular model (or set of models).

Re: Gemini 2.5 Pro Preview

#229

Post by cubefox »

jjani wrote: post #183
sauwan wrote: post #151

Earlier quoted context omitted.

I'd be surprised if this was a new base model. It sounds like they just did some post-training RL tuning to make this version specifically stronger for coding, at the expense of other priorities.

Every frontier model now is a distill of a larger unpublished model. This could be a slightly smaller distill, with potentially the extra tuning you're mentioning.

That's an unsubstantiated claim. I doubt this is true, since people are disproportionately more willing to pay for the best of the best, rather than for something worse.

Re: Gemini 2.5 Pro Preview

#230

Post by ajross »

Volundr wrote: post #165
redox99 wrote: post #92

Earlier quoted context omitted.

Making LLMs know what they don't know is a hard problem. Many attempts at making them refuse to answer what they don't know caused them to refuse to answer things they did in fact know.

> Many attempts at making them refuse to answer what they don't know caused them to refuse to answer things they did in fact know. Are we sure they know these things as opposed to being able to consistently guess correctly? With LLMs I'm not sure we even have a clear definition of what it means for it to "know" something.

> Are we sure they know these things as opposed to being able to consistently guess correctly?

What is the practical difference you're imagining between "consistently correct guess" and "knowledge"?

LLMs aren't databases. We have databases. LLMs are probabilistic inference engines. All they do is guess, essentially. The discussion here is about how to get the guess to "check itself" with a firmer idea of "truth". And it turns out that's hard because it requires that the guessing engine know that something needs to be checked in the first place.

Post reply on HN