181–190 of 728 posts
Earlier quoted context omitted.
I expect my calculator to be 100% accurate 100% of the time. I have slightly more tolerance for other software having defects, but not much more.
I don't think that's the relevant comparison though. Do you expect StackOverflow or product documentation to be 100% accurate 100% of the time? I definitely don't.
Earlier quoted context omitted.
I expect my calculator to be 100% accurate 100% of the time. I have slightly more tolerance for other software having defects, but not much more.
Are you sure about that? Try these.. - (1e(1e10) + 1) - 1e(1e10) - sqrt(sqrt(2)) * sqrt(sqrt(2)) * sqrt(sqrt(2)) * sqrt(sqrt(2))
Earlier quoted context omitted.
Sounds like they were losing so much money on 2.5-Pro they came up with a forced update that made it cheaper to run. They can't come out with "we've made it worse across the board", nor do they want to be the first to actually raise prices, so instead they made a bit of a distill that's slightly better at coding so they can still spin it positively.
I'd be surprised if this was a new base model. It sounds like they just did some post-training RL tuning to make this version specifically stronger for coding, at the expense of other priorities.
Earlier quoted context omitted.
I expect my calculator to be 100% accurate 100% of the time. I have slightly more tolerance for other software having defects, but not much more.
And a $2.99 drugstore slim wallet calculator with solar power gets it right 100% of the time while billion dollar LLMs can still get arithmetic wrong on occasion.
My frustration with using these models for programming in the past has largely been around their tendency to hallucinate APIs that simply don't exist. The Gemini 2.5 models, both pro and flash, seem significantly less susceptible to this than any other model I've tried. There are still significant limitations, no amount of prompting will get current models to approach abstraction and architecture the way a person doe…
Never seen it fumble that around
Swear people act like humans themselves don’t ever need to be asked for clarification
Earlier quoted context omitted.
I'm seeing it trying to catch blind exceptions in Python all the time. I see it in my colleagues code all the time, it's driving me nuts.
There are a bunch of stupid behaviors of LLM coding that will be fixed by more awareness pretty soon. Imagine putting the docs and code for all of your libraries into the context window so it can understand what exceptions might be thrown!
Earlier quoted context omitted.
Making LLMs know what they don't know is a hard problem. Many attempts at making them refuse to answer what they don't know caused them to refuse to answer things they did in fact know.
> Many attempts at making them refuse to answer what they don't know caused them to refuse to answer things they did in fact know. Are we sure they know these things as opposed to being able to consistently guess correctly? With LLMs I'm not sure we even have a clear definition of what it means for it to "know" something.
But also things where guessing was desirable. For example with a riddle it would tell you it did not know or there wasn't enough information. After pressuring it to answer anyway it would correctly solve the riddle.
The official llama 2 finetune was pretty bad with this stuff.
Interestingly, when compering benchmarks of Experimental 03-25 [1] and Experimental 05-06 [2] it seems the new version scores slightly lower in everything except on LiveCodeBench. [1] https://storage.googleapis.com/model-cards/documents/gemini-... [2] https://deepmind.google/technologies/gemini/