Live data from Hacker News

Gemini 2.5 Pro Preview

developers.googleblog.com

331–340 of 728 posts

Re: Gemini 2.5 Pro Preview

#331

My frustration with using these models for programming in the past has largely been around their tendency to hallucinate APIs that simply don't exist. The Gemini 2.5 models, both pro and flash, seem significantly less susceptible to this than any other model I've tried. There are still significant limitations, no amount of prompting will get current models to approach abstraction and architecture the way a person doe…

I don't know about that, my own adventures with Gemini Pro 2.5 in Roo Code has it outputting code in a style that is very close to my own

While far from perfect for large projects, controlling the scope of individual requests (with orchestrator/boomerang mode, for example) seems to do wonders

Given the sheer, uh, variety of code I see day to day in an enterprise setting, maybe the problem isn't with Gemini?

Re: Gemini 2.5 Pro Preview

#333

My frustration with using these models for programming in the past has largely been around their tendency to hallucinate APIs that simply don't exist. The Gemini 2.5 models, both pro and flash, seem significantly less susceptible to this than any other model I've tried. There are still significant limitations, no amount of prompting will get current models to approach abstraction and architecture the way a person doe…

I feel like there are two realities right now where half the people say LLM doesn't do anything well and there is another half that's just using LLM to the max. Can everybody preface what stack they are using or what exactly they are doing so we can better determine why it's not working for you? Maybe even include what your expectations are? Maybe even tell us what models you're using? How are you prompting the models exactly?

I find for 90% of the things I'm doing LLM removes 90% of the starting friction and let me get to the part that I'm actually interested in. Of course I also develop professionally in a python stack and LLMs are 1 shotting a ton of stuff. My work is standard data pipelines and web apps.

I'm a tech lead at faang adjacent w/ 11YOE and the systems I work with are responsible for about half a billion dollars a year in transactions directly and growing. You could argue maybe my standards are lower than yours but I think if I was making deadly mistakes the company would have been on my ass by now or my peers would have caught them.

Everybody that I work with is getting valuable output from LLMs. We are using all the latest openAI models and have a business relationship with openAI. I don't think I'm even that good at prompting and mostly rely on "vibes". Half of the time I'm pointing the model to an example and telling it "in the style of X do X for me".

I feel like comments like these almost seem gaslight-y or maybe there's just a major expectation mismatch between people. Are you expecting LLMs to just do exactly what you say and your entire job is to sit back prompt the LLM? Maybe I'm just use to shit code but I've looked at many code bases and there is a huge variance in quality and the average is pretty poor. The average code that AI pumps out is much better.

Re: Gemini 2.5 Pro Preview

#334
post #219

Earlier quoted context omitted.

I’ve seen benchs on hallucinations and OpenAI has typically performed worse than Google and Anthropic models. Sometimes significantly so. But it doesn’t seem like they have cared much. I’ve suspected that LLM performance is correlated to risking hallucinations? That is, if they’re bolder, this can be beneficial? Which helps in other performance benchmarks. But of course at the risk of hallucinating more…

The hallucinations are a result of RLVR. We reward the model for an answer and then force it to reason about how to get there when the base model may not have that information.

> The hallucinations are a result of RLVR

Well let us reward them for producing output that is consistent with database accessed selected documentation then, and massacre them for output they cannot justify - like we do with humans.

Re: Gemini 2.5 Pro Preview

#336

My frustration with using these models for programming in the past has largely been around their tendency to hallucinate APIs that simply don't exist. The Gemini 2.5 models, both pro and flash, seem significantly less susceptible to this than any other model I've tried. There are still significant limitations, no amount of prompting will get current models to approach abstraction and architecture the way a person doe…

> no amount of prompting will get current models to approach abstraction and architecture the way a person does I find this sentiment increasingly worrisome. It's entirely clear that every last human will be beaten on code design in the upcoming years (I am not going to argue if it's 1 or 5 years away, who cares?) I wished people would just stop holding on to what amounts to nothing, and think and talk more about wha…

I mean, didn't you just admit you are wrong? If we are talking 1-5 years out, that's not "current models".

Re: Gemini 2.5 Pro Preview

#337

My frustration with using these models for programming in the past has largely been around their tendency to hallucinate APIs that simply don't exist. The Gemini 2.5 models, both pro and flash, seem significantly less susceptible to this than any other model I've tried. There are still significant limitations, no amount of prompting will get current models to approach abstraction and architecture the way a person doe…

This is hilarious to read if you have actually seen the average (embedded systems) production code written by humans.

Either you have no idea how terrible real world commercial software (architecture) is or you're vastly underestimating newer LLMs or both.

Re: Gemini 2.5 Pro Preview

#338

My frustration with using these models for programming in the past has largely been around their tendency to hallucinate APIs that simply don't exist. The Gemini 2.5 models, both pro and flash, seem significantly less susceptible to this than any other model I've tried. There are still significant limitations, no amount of prompting will get current models to approach abstraction and architecture the way a person doe…

I've had great success by asking it to do project design first, compose the design into an artifact, and then asking it to consult the design artifact as it writes code.

This is a great idea - do you have a more detailed overview of this approach and/or an example? What types of things do you tell it to put into the "artefact"?

Re: Gemini 2.5 Pro Preview

#339

Usually don’t believe the benchmarks but first in web dev arena specifically is crazy. That one has been Claude for so long, which tracks in my experience

I'm wondering when Claude 4 will drop. It's long overdue.

For me, Claude 3.7 was a noticeable step down across a wide range of tasks when compared to 3.5 with the same prompt. Benchmarks are one thing, but for real life use, I kept finding myself switching back to 3.5. Wouldn't be surprised if they were trying to figure out what happened there and how to prevent that in the next version.

Re: Gemini 2.5 Pro Preview

#340
post #40

Earlier quoted context omitted.

I have been asking if AI without hallucination, coding or not is possible but so far with no real concrete answer.

It's already much improved on the early days. But I wonder when we'll be happy? Do we expect colleagues friends and family to be 100% laser-accurate 100% of the time? I'd wager we don't. Should we expect that from an artificial intelligence too?

Yes we want people "in the game" to be of sound mind. (The matter there is not about being accurate, but of being trustworthy - substance, not appearance.)

And tools in the game, even more so (there's no excuse for the engineered).

Post reply on HN