Gemini 2.5 Pro Preview • Hacker News Board

Re: Gemini 2.5 Pro Preview

#381

Post by cma » Tue, May 06, 2025, 9:02 PM UTC

arnaudsm wrote: post #94

andy12_ wrote: post #74

Interestingly, when compering benchmarks of Experimental 03-25 [1] and Experimental 05-06 [2] it seems the new version scores slightly lower in everything except on LiveCodeBench. [1] https://storage.googleapis.com/model-cards/documents/gemini-... [2] https://deepmind.google/technologies/gemini/

This should be the top comment. Cherry-picking is hurting this industry. I bet they kept training on coding tasks, made everything worse on the way, and tried to hide it under the rug because of the sunk costs.

They likely knew continued training on code would have some amount of catastrophic forgetting on other stuff. They didn't throw away the old weights so probably not sunk cost fallacy going on, but since it is relatively new and they found out X% of API token spend was on coding agents (where X is huge), compared to what token spend distribution looked like on prior Geminis that couldn't code well, they probably didn't want the complexity and worse batching of having another model for it if the impacts weren't too large and decided they didn't weight coding enough initially and it is worth the tradeoffs.

Re: Gemini 2.5 Pro Preview

#382

Post by epolanski » Tue, May 06, 2025, 9:02 PM UTC

aero142 wrote: post #354

paulirish wrote: post #250

> Gemini 2.5 Pro now ranks #1 on the WebDev Arena leaderboard It'd make sense to rename WebDev Arena to React/Tailwind Arena. Its system prompt requires [1] those technologies and the entire tool breaks when requesting vanilla JS or other frameworks. The second-order implications of models competing on this narrow definition of webdev are rather troublesome. [1] https://blog.lmarena.ai/blog/2025/webdev-arena/#:~:text…

If llms are able to write better code with more declarative and local programming components and tailwind, then I could imagine a future where a new programming language is created to maximize llm success.

This so much.

To me it seems so strange that few good language designers and ml folks didn't group together to work on this.

It's clear that there is a space for some LLM meta language that could be designed to compile to bytecode, binary, JS, etc.

It also doesn't need to be textual like we code, but some form of AST llama can manipulate with ease.

Re: Gemini 2.5 Pro Preview

#383

Post by ramoz » Tue, May 06, 2025, 9:05 PM UTC

ionwake wrote: post #190

Can someone tell me if windsurf is better than cursor? ( pref someone who has used both for a few days? )

Claude Code and its not close. I feed my entire project to gemini for planning and figuring out complex solutions for claude code to execute on. I use Prompt Tower for building entire codebase prompts for gemini.

Re: Gemini 2.5 Pro Preview

#384

Post by epolanski » Tue, May 06, 2025, 9:11 PM UTC

obsolete_wagie wrote: post #188

o3 is so far ahead of antrhopic and google, these models arent even worth using

Not my experience, at all.

I have long stopped using OpenAI products, and all oX have been letdowns.

For coding it has been Claude 3.5 -> 3.7 -> Gemini 2.5 for me. For general use it has been chatgpt -> Gemini.

Google has retaken the ML crown for my use cases and it keeps getting better.

Gemini 2.0 flash was also the first LLM I put in production, because for my use case (summarizing news articles and translate them) it was way too fast, accurate and cheap to ignore whereas ChatGPT was consistently too slow and expensive to be even considered.

Re: Gemini 2.5 Pro Preview

#385

Post by wewewedxfgdf » Tue, May 06, 2025, 9:11 PM UTC

krat0sprakhar wrote: post #375

wewewedxfgdf wrote: post #373

Gemini does not accept upload of TSX files, it says "File type unsupported" You must rename your files to .tsx.txt THEN IT ACCEPTS THEM and works perfectly fine writing TSX code. This is absolutely bananas. How can such a powerful coding engine have this behavior?

Where are you testing this? I'm able to upload tsx files on aistudio

https://gemini.google.com/app

Re: Gemini 2.5 Pro Preview

#386

Post by NeutralForest » Tue, May 06, 2025, 9:11 PM UTC

JoshuaDavid wrote: post #233

NeutralForest wrote: post #125

Earlier quoted context omitted.

I'm seeing it trying to catch blind exceptions in Python all the time. I see it in my colleagues code all the time, it's driving me nuts.

The training loop asked the model to one-shot working code for the given problems without being able to iterate. If you had to write code that had to work on the first try, and where a partially correct answer was better than complete failure, I bet your code would look like that too. In any case, it knows what good code looks like. You can say "take this code and remove spurious comments and prefer narrow exception…

It's only an example, there's pretty of irrelevant stuff that LLMs default to which is pretty bad Python. I'm not saying it's always bad but there's a ton of not so nice code or subtly wrong code generated (for example file and path manipulation).

Re: Gemini 2.5 Pro Preview

#387

Post by NeutralForest » Tue, May 06, 2025, 9:13 PM UTC

jerkstate wrote: post #139

NeutralForest wrote: post #125

Earlier quoted context omitted.

I'm seeing it trying to catch blind exceptions in Python all the time. I see it in my colleagues code all the time, it's driving me nuts.

There are a bunch of stupid behaviors of LLM coding that will be fixed by more awareness pretty soon. Imagine putting the docs and code for all of your libraries into the context window so it can understand what exceptions might be thrown!

It just feels to me like trying to derive correct behavior without a proper spec so I don't see how it'll get that much better. Maybe we'll collectively remove the pathological code but otherwise I'm not seeing it.

Re: Gemini 2.5 Pro Preview

#388

Post by thewebguyd » Tue, May 06, 2025, 9:13 PM UTC

abletonlive wrote: post #333

segphault wrote: post #18

My frustration with using these models for programming in the past has largely been around their tendency to hallucinate APIs that simply don't exist. The Gemini 2.5 models, both pro and flash, seem significantly less susceptible to this than any other model I've tried. There are still significant limitations, no amount of prompting will get current models to approach abstraction and architecture the way a person doe…

I feel like there are two realities right now where half the people say LLM doesn't do anything well and there is another half that's just using LLM to the max. Can everybody preface what stack they are using or what exactly they are doing so we can better determine why it's not working for you? Maybe even include what your expectations are? Maybe even tell us what models you're using? How are you prompting the model…

I've found, like you mentioned, that the tech stack you work with matters a lot in terms of successful results from LLMs.

Python is generally fine, as you've experienced, as is JavaScript/TypeScript & React.

I've had mixed results with C# and PowerShell. With PowerShell, hallucinations are still a big problem. Not sure if it's the Noun-Verb naming scheme of cmdlets, but most models still make up cmdlets that don't exist on the fly (though will correct itself once you correct it that it doesn't exist but at that point - why bother when I can just do it myself correctly the first time).

With C#, even with my existing code as context, it can't adhere to a consistent style, and can't handle nullable reference types (albeit, a relatively new feature in C#). It works, but I have to spend too much time correcting it.

Given my own experiences and the stacks I work with, I still won't trust an LLM in agent mode. I make heavy use of them as a better Google, especially since Google has gone to shit, and to bounce ideas off of, but I'll still write the code myself. I don't like reviewing code, and having LLMs write code for me just turns me into a full time code reviewer, not something I'm terribly interested in becoming.

I still get a lot of value out of the tools, but for me I'm still hesitant to unleash them on my code directly. I'll stick with the chat interface for now.

edit Golang is another language I've had problems relying on LLMs for. On the flip side, LLMs have been great for me with SQL and I'm grateful for that.

Re: Gemini 2.5 Pro Preview

#389

Post by thewebguyd » Tue, May 06, 2025, 9:24 PM UTC

mattlondon wrote: post #308

kweingar wrote: post #64

Earlier quoted context omitted.

I expect my calculator to be 100% accurate 100% of the time. I have slightly more tolerance for other software having defects, but not much more.

AIs aren't intended to be used as calculators though? You could say that when I use my spanner/wrench to tighten a nut it works 100% of the time, but as soon as I try to use a screwdriver it's terrible and full of problems and it can't even reliably so something as trivially easy as tighten a nut, even though a screwdriver works the same way by using torque to tighten a fastener. Well that's because one tool is desig…

> AIs aren't intended to be used as calculators though?

Then why are we using them to write code, which should produce reliable outputs for a given input...much like a calculator.

Obviously we want the code to produce correct results for whatever input we give, and as it stands now, I can't trust LLM output without reviewing first. Still a helpful tool, but ultimately my desire would be to have them be as accurate as a calculator so they can be trusted enough to not need the review step.

Using an LLM and being OK with untrustworthy results, it'd be like clicking the terminal icon on my dock and sometimes it opens terminal, sometimes it might open a browser, or just silently fail because there's no reproducible output for any given input to an LLM. To me that's a problem, output should be reproducible, especially if it's writing code.

Re: Gemini 2.5 Pro Preview

#390

Post by Volundr » Tue, May 06, 2025, 9:27 PM UTC

ajross wrote: post #230

Volundr wrote: post #165

Earlier quoted context omitted.

> Many attempts at making them refuse to answer what they don't know caused them to refuse to answer things they did in fact know. Are we sure they know these things as opposed to being able to consistently guess correctly? With LLMs I'm not sure we even have a clear definition of what it means for it to "know" something.

> Are we sure they know these things as opposed to being able to consistently guess correctly? What is the practical difference you're imagining between "consistently correct guess" and "knowledge"? LLMs aren't databases. We have databases. LLMs are probabilistic inference engines. All they do is guess, essentially. The discussion here is about how to get the guess to "check itself" with a firmer idea of "truth". And…

> What is the practical difference you're imagining between "consistently correct guess" and "knowledge"?

Knowing it's correct. You've just instructed it not to guess remember? With practice people can get really good at guessing all sorts of things.

I think people have a serious misunderstanding about how these things work. They don't have their training set sitting around for reference. They are usually guessing. Most of the time with enough consistency that it seems like they "know'. Then when they get it wrong we call it "hallucinations". But instructing then not to guess means suddenly they can't answer much. There no guessing vs not with an LLM, it's all the same statistical process, the difference is just if it gives the right answer or not.