Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 2.5 Pro Preview

developers.googleblog.com

Post reply on HN
Print view View original

171–180 of 728 posts

Re: Gemini 2.5 Pro Preview

#171

Post by doug_durham »

segphault wrote: post #18

My frustration with using these models for programming in the past has largely been around their tendency to hallucinate APIs that simply don't exist. The Gemini 2.5 models, both pro and flash, seem significantly less susceptible to this than any other model I've tried. There are still significant limitations, no amount of prompting will get current models to approach abstraction and architecture the way a person doe…

If they never get good at abstraction or architecture they will still provide a tremendous amount of value. I have them do the parts of my job that I don't like. I like doing abstraction and architecture.

Re: Gemini 2.5 Pro Preview

#172

Post by jtwaleson »

cryptoz wrote: post #100
mgw wrote: post #72

Earlier quoted context omitted.

This has also been my biggest gripe with Gemini 2.5 Pro. While it is fantastic at one-shotting major new features, when wanting to make smaller iterative changes, it always does big refactors at the same time. I haven't found a way to change that behavior through changes in my prompts. Claude 3.7 Sonnet is much more restrained and does smaller changes.

This exact problem is something I’m hoping to fix with a tool that parses the source to AST and then has the LLM write code to modify the AST (which you then run to get your changes) rather than output code directly. I’ve started in a narrow niche of python/flask webapps and constrained to that stack for now, but if you’re interested I’ve just opened it for signups: https://codeplusequalsai.com Would love feedback! E…

Having the LLM modify the AST seems like a great idea. Constraining an LLM to only generate valid code would be super interesting too. Hope this works out!

Re: Gemini 2.5 Pro Preview

#173

Post by [deleted] »

mattlondon wrote: post #46
ksec wrote: post #40

Earlier quoted context omitted.

I have been asking if AI without hallucination, coding or not is possible but so far with no real concrete answer.

It's already much improved on the early days. But I wonder when we'll be happy? Do we expect colleagues friends and family to be 100% laser-accurate 100% of the time? I'd wager we don't. Should we expect that from an artificial intelligence too?

[deleted]

Re: Gemini 2.5 Pro Preview

#174

Post by pizza »

kweingar wrote: post #64
mattlondon wrote: post #46

Earlier quoted context omitted.

It's already much improved on the early days. But I wonder when we'll be happy? Do we expect colleagues friends and family to be 100% laser-accurate 100% of the time? I'd wager we don't. Should we expect that from an artificial intelligence too?

I expect my calculator to be 100% accurate 100% of the time. I have slightly more tolerance for other software having defects, but not much more.

Are you sure about that? Try these..

- (1e(1e10) + 1) - 1e(1e10)

- sqrt(sqrt(2)) * sqrt(sqrt(2)) * sqrt(sqrt(2)) * sqrt(sqrt(2))

Re: Gemini 2.5 Pro Preview

#175

Post by Tainnor »

segphault wrote: post #18

My frustration with using these models for programming in the past has largely been around their tendency to hallucinate APIs that simply don't exist. The Gemini 2.5 models, both pro and flash, seem significantly less susceptible to this than any other model I've tried. There are still significant limitations, no amount of prompting will get current models to approach abstraction and architecture the way a person doe…

I definitely get more use out of Gemini Pro than other models I've tried, but it's still very prone to bullshitting.

I asked it a complicated question about the Scala ZIO framework that involved subtyping, type inference, etc. - something that would definitely be hard to figure out just from reading the docs. The first answer it gave me was very detailed, very convincing and very wrong. Thankfully I noticed it myself and was able to re-prompt it and I got an answer that is probably right. So it was useful in the end, but only because I realised that the first answer was nonsense.

Re: Gemini 2.5 Pro Preview

#176

Post by tclancy »

NeutralForest wrote: post #125
koakuma-chan wrote: post #55

Earlier quoted context omitted.

No, it's just bad. I've been writing a lot of Python code past two days with Gemini 2.5 Pro Preview, and all of its code was like: ```python def whatever(): --- SECTION ONE OF THE CODE --- ... --- SECTION TWO OF THE CODE --- try: [some "dangerous" code] except Exception as e: logging.error(f"Failed to save files to {output_path}: {e}") # Decide whether to raise the error or just warn # raise IOError(f"Failed to save…

I'm seeing it trying to catch blind exceptions in Python all the time. I see it in my colleagues code all the time, it's driving me nuts.

Well, at least now we know who to blame for the training data :)

Re: Gemini 2.5 Pro Preview

#177

Post by panarchy »

scrlk wrote: post #157
panarchy wrote: post #154

Is it just me that finds that while Gemini 2.5 is able to generate a lot of code that the end results are usually lackluster compared to Claude and even ChatGPT? I also find it hard-headed and frequently does things in ways I explicitly told it not to. The massive context window is pretty great though and enables me to do things I can't with the others so it still gets used a lot.

How are you using it? I find that I get the best results from 2.5 Pro via Google AI Studio with a low temperature (0.2-0.3).

AI Studio as well, but I haven't played around with the temperature too much and even then I only lowered it to like 0.8 a few times. So I'll have to try this out. Thanks.

Re: Gemini 2.5 Pro Preview

#178

Post by scbenet »

childintime wrote: post #159

How does it perform on anything but Python and Javascript? In my experience my milage varied a lot when using C#, for example, or Zig, so I've learnt to just let it select the language it wants. Also, why doesn't Ctrl+C work??

It's very good at Go, which makes sense because I'm assuming it's trained on a lot of Google's code

Re: Gemini 2.5 Pro Preview

#179

Post by ctxc »

Analemma_ wrote: post #169
kweingar wrote: post #64

Earlier quoted context omitted.

I expect my calculator to be 100% accurate 100% of the time. I have slightly more tolerance for other software having defects, but not much more.

I don't think that's the relevant comparison though. Do you expect StackOverflow or product documentation to be 100% accurate 100% of the time? I definitely don't.

The error introduced by the data is expected and internalized, it's the error of LLMs on _top_ of that that's hard to.

Re: Gemini 2.5 Pro Preview

#180

Post by scarab92 »

kortilla wrote: post #119
mattlondon wrote: post #46

Earlier quoted context omitted.

It's already much improved on the early days. But I wonder when we'll be happy? Do we expect colleagues friends and family to be 100% laser-accurate 100% of the time? I'd wager we don't. Should we expect that from an artificial intelligence too?

If colleagues lie with the certainty that LLMs do, they would get fired for incompetence.

I wish that were true, but I’ve found that certain types of employees do confidently lie as much as llms, especially when answering “do you understand” type questions
Post reply on HN