Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 2.5 Pro Preview

developers.googleblog.com

Post reply on HN
Print view View original

301–310 of 728 posts

Re: Gemini 2.5 Pro Preview

#301

Post by kweingar »

Analemma_ wrote: post #169
kweingar wrote: post #64

Earlier quoted context omitted.

I expect my calculator to be 100% accurate 100% of the time. I have slightly more tolerance for other software having defects, but not much more.

I don't think that's the relevant comparison though. Do you expect StackOverflow or product documentation to be 100% accurate 100% of the time? I definitely don't.

I actually agree with this. I use LLMs often, and I don't compare them to a calculator.

Mainly I meant to push back against the reflexive comparison to a friend or family member or colleague. AI is a multi-purpose tool that is used for many different kinds of tasks. Some of these tasks are analogues to human tasks, where we should anticipate human error. Others are not, and yet we often ask an LLM to do them anyway.

Re: Gemini 2.5 Pro Preview

#302

Post by xd1936 »

martinsnow wrote: post #279
paulirish wrote: post #250

> Gemini 2.5 Pro now ranks #1 on the WebDev Arena leaderboard It'd make sense to rename WebDev Arena to React/Tailwind Arena. Its system prompt requires [1] those technologies and the entire tool breaks when requesting vanilla JS or other frameworks. The second-order implications of models competing on this narrow definition of webdev are rather troublesome. [1] https://blog.lmarena.ai/blog/2025/webdev-arena/#:~:text…

Bwoah it's almost as if react and tailwind is the bees knees ind frontend atm

[flagged]

Re: Gemini 2.5 Pro Preview

#303

Post by ramesh31 »

Workaccount2 wrote: post #232
kweingar wrote: post #99

Earlier quoted context omitted.

If it's zero effort, then why do devs need to adapt fast? And wouldn't adapting be incredibly easy? The only disadvantage to not using these tools would be that your current output is slower. As soon as your employer asks for more or you're looking for a new job, you can just turn on AI and be as fast as everyone who already uses it.

"Why are we paying you $150k/yr to middleman a chatbot?"

>"Why are we paying you $150k/yr to middleman a chatbot?"

Because I don't get paid $150k/yr to write HTML and CSS. I get paid to provide technical solutions to business problems. And "chatbots" are a very useful new tool to aid in that.

Re: Gemini 2.5 Pro Preview

#304

Post by zurfer »

merksittich wrote: post #143
andy12_ wrote: post #74

Interestingly, when compering benchmarks of Experimental 03-25 [1] and Experimental 05-06 [2] it seems the new version scores slightly lower in everything except on LiveCodeBench. [1] https://storage.googleapis.com/model-cards/documents/gemini-... [2] https://deepmind.google/technologies/gemini/

According to the article, "[t]he previous iteration (03-25) now points to the most recent version (05-06)." I assume this applies to both the free tier gemini-2.5-pro-exp-03-25 in the API (which will be used for training) and the paid tier gemini-2.5-pro-preview-03-25. Fair enough, one could say, as these were all labeled as preview or experimental. Still, considering that the new model is slightly worse across the b…

Just switching a pinned version (even alpha, beta, experimental, preview) to another model doesn't feel right.

I get it, chips are sparse and they want their capacity back, but it breaks trust with developers to just downgrade your model.

Call it gemini-latest and I understand that things will change. Call it *-03-25 and I want the same model that I got on 25th March.

Re: Gemini 2.5 Pro Preview

#305

Post by disgruntledphd2 »

yousif_123123 wrote: post #292
segphault wrote: post #18

My frustration with using these models for programming in the past has largely been around their tendency to hallucinate APIs that simply don't exist. The Gemini 2.5 models, both pro and flash, seem significantly less susceptible to this than any other model I've tried. There are still significant limitations, no amount of prompting will get current models to approach abstraction and architecture the way a person doe…

The opposite problem is also true. I was using it to edit code I had that was calling the new openai image API, which is slightly different from the dalle API. But Gemini was consistently "fixing" the OpenAI call even when I explained clearly not to do that since I'm using a new API design etc. Claude wasn't having that issue. The models are very impressive. But issues like these still make me feel they are still mor…

They are definitely pattern matching. Like, that's how we train them, and no matter how many layers of post training you add, you won't get too far from next token prediction.

And that's fine and useful.

Re: Gemini 2.5 Pro Preview

#306

Post by disgruntledphd2 »

toomuchtodo wrote: post #300
yousif_123123 wrote: post #292

Earlier quoted context omitted.

The opposite problem is also true. I was using it to edit code I had that was calling the new openai image API, which is slightly different from the dalle API. But Gemini was consistently "fixing" the OpenAI call even when I explained clearly not to do that since I'm using a new API design etc. Claude wasn't having that issue. The models are very impressive. But issues like these still make me feel they are still mor…

It seems like the fix is straightforward (check the output against a machine readable spec before providing it to the user), but perhaps I am a rube. This is no different than me clicking through a search result to the underlying page to verify the veracity of the search result surfaced.

Why coding agents et al don't make use of the AST through LSP is a question I've been asking myself since the first release of GitHub copilot.

I assume that it's trickier than it seems as it hasn't happened yet.

Re: Gemini 2.5 Pro Preview

#307

Post by mannycalavera42 »

segphault wrote: post #18

My frustration with using these models for programming in the past has largely been around their tendency to hallucinate APIs that simply don't exist. The Gemini 2.5 models, both pro and flash, seem significantly less susceptible to this than any other model I've tried. There are still significant limitations, no amount of prompting will get current models to approach abstraction and architecture the way a person doe…

same, I asked a simple question about javascript fetch api and it started talking about the workspace api. When I asked about that workspace api it replied it was the google workspace API ¯ \ _ (ツ) _ / ¯

Re: Gemini 2.5 Pro Preview

#308

Post by mattlondon »

kweingar wrote: post #64
mattlondon wrote: post #46

Earlier quoted context omitted.

It's already much improved on the early days. But I wonder when we'll be happy? Do we expect colleagues friends and family to be 100% laser-accurate 100% of the time? I'd wager we don't. Should we expect that from an artificial intelligence too?

I expect my calculator to be 100% accurate 100% of the time. I have slightly more tolerance for other software having defects, but not much more.

AIs aren't intended to be used as calculators though?

You could say that when I use my spanner/wrench to tighten a nut it works 100% of the time, but as soon as I try to use a screwdriver it's terrible and full of problems and it can't even reliably so something as trivially easy as tighten a nut, even though a screwdriver works the same way by using torque to tighten a fastener.

Well that's because one tool is designed for one thing, and one is designed for another.

Re: Gemini 2.5 Pro Preview

#310

Post by mvdtnz »

I truly do not understand how people are getting worthwhile results from Gemini 2.5 Pro. I have used all of the major models for lots of different programming tasks and I have never once had Gemini produce something useful. It's not just wrong, it's laughably bad. And people are making claims that it's the best. I just... don't... get it.
Post reply on HN