Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 3.1 Pro

blog.google

Post reply on HN
Print view View original

911–920 of 951 posts

Re: Gemini 3.1 Pro

#911

Post by tempestn »

holbrad wrote: post #901
maxwellcoffee wrote: post #766

Earlier quoted context omitted.

Google actually has the BEST ratings in the AA-Omniscience Index: AA-Omniscience Index (higher is better) measures knowledge reliability and hallucination. It rewards correct answers, penalizes hallucinations, and has no penalty for refusing to answer. Gemini 3.1 is the top spot, followed by 3.0 and then opus 4.6 max

This isn't actually correct. Gemini 3.0 gets a very high score because it's very often correct, but it does not have a low hallucination rate. https://artificialanalysis.ai/#aa-omniscience-hallucination-... It looks like 3.1 is a big improvement in this regard, it hallucinates a lot less.

Yes and no. The hallucination rate shown there is the percentage of time the model answers incorrectly when it should have instead admitted to not knowing the answer. Most models score very poorly on this, with a few exceptions, because they nearly always try to answer. It's true that 3.0 is no better than others on this. By given that it does know the correct answers much more often than eg. GPT 5.2, it does in fact give hallucinated answers much less often.

In short, its hallucination rate as a percentage of unknown answers is no better than most models, but its hallucination rate as a percentage of total answers in indeed better.

Re: Gemini 3.1 Pro

#912

Post by camel_Snake »

startages wrote: post #896
sdeiley wrote: post #548

People underrate Google's cost effectiveness so much. Half price of Opus. HALF. Think about ANY other product and what you'd expect from the competition thats half the price. Yet people here act like Gemini is dead weight ____ Update: 3.1 was 40% of the cost to run AA index vs Opus Thinking AND SONNET, beat Opus, and still 30% faster for output speed. https://artificialanalysis.ai/?speed=intelligence-vs-speed&m...

This is misleading. I'm running a live experiment here: https://project80.divcrafts.com/ There are 4 models, all receiving the exact same prompts a few times a day, required to respond with a specific action. In the first experiment I used gemini-3-pro-preview, it spent ~$18 on the same task where Opus 4.5 spent ~$4, GPT-5.1 spent ~$4.50, and Grok spent ~$7. Pro was burning through money so fast I switched to gemini-…

Is that no longer the case, or am I misunderstanding the operational costs displayed?

Opus: 521k input tokens; 12k out

Grok: 443k input tokens; 57k out

Gemini: 677k input tokens; 7k out

OAI: 543k input tokens; 17k out

Gemini appears to use by far the least amount of reasoning tokens, assuming they're included in the output counts.

Re: Gemini 3.1 Pro

#913

Post by conception »

My current AI test. There was a BBS I was on in the 90s and there was this door game I hadn't seen anywhere else. I simply describe the BBS, where it was popular, its name, the year it was around, and the BBS game and a description of it mechanics, etc.

OpenAI and Google's Deep Research produce a very long, 100% made up report. If I question the AI on the report, they both admit they just made it up.

Claude just returns, "I couldn't find anything on the BBS or the game."

Re: Gemini 3.1 Pro

#914

Post by quotemstr »

kees99 wrote: post #415
quotemstr wrote: post #353

Earlier quoted context omitted.

I wouldn't even think of letting an agent work in that made. Even the best of them produce garbage code unless I keep them on a tight leash. And no, not a skill issue. What I don't have time to do is debug obvious slop.

I ended up running codex with all the "danger" flags, but in a throw-away VM with copy-on-write access to code folders. Built-in approval thing sounds like a good idea, but in practice it's unusable. Typical session for me was like: About to run "sed -n '1,100p' example.cpp", approve? About to run "sed -n '100,200p' example.cpp", approve? About to run "sed -n '200,300p' example.cpp", approve? Could very well be a ski…

One decent approach (which Codex implements, and some others) is to run these commands in a real-only sandbox without approval and let the model ask your approval when it wants to run outside the sandbox. An even better approach is just doing abstract interpretation over shell command proposals.

You want something like codex -a read-only -s on-failure (from memory: look up the exact flags)

Re: Gemini 3.1 Pro

#915

Post by barfingclouds »

I’m no tech expert like a lot of people here, but I find Gemini 3.0 insanely good for my regular daily questions. Hoping this one is great too. I’m kind of at the point where many answers are essentially perfect and I don’t know if I need much more

Re: Gemini 3.1 Pro

#916

Post by andai »

bluegatty wrote: post #652
andai wrote: post #611

Earlier quoted context omitted.

Tell me more about Codex. I'm trying to understand it better. I have a pretty crude mental model for this stuff but Opus feels more like a guy to me, while Codex feels like a machine. I think that's partly the personality and tone, but I think it goes deeper than that. (Or maybe the language and tone shapes the behavior, because of how LLMs work? It sounds ridiculous but I told Claude to believe in itself and suddenl…

Your intuition is exactly correct - it's not just 'tone' it's 'deeper than that'. Codex is a 'poor communicator' - which matters surprisingly a lot in these things. It's overly verbose, it often misses the point - but - it is slightly stronger in some areas. Also - Codex now has 'Spark' which is on Cerebras, it's wildly fast - and this absolutely changes 'workflow' fundamentally. With 'wait-thinking' - you an have 3-…

>human is the limiting factor

I read an article recently, "starting to feel like I'm the one holding the AI back" and that stayed with me... I think that's true both individually and collectively. Ostensibly we're aiming for self-improvement, but there's explicit training against it, for various reasons...

Try asking Opus about Living Information Systems and see if you get the same result I did!

Re: Gemini 3.1 Pro

#917

Post by bluegatty »

embedding-shape wrote: post #862
bluegatty wrote: post #704

Earlier quoted context omitted.

You're right. It's funny because I kind of noticed that, but with all of these subtle model issues, I'm so used to being distraught by the smallest thing I've had to learn to 'trust the data' aka the charts, model standings, performance, etc. and in this case, I was under the assumption 'it was the same model' clearly it's not. Which is a bummer because it would be nice to try a true side-by-side analysis.

> It's funny because I kind of noticed that It's less funny when you consider that you were very confident about it, yet now it seems you haven't even bothered to run the model yourself, as you'd notice how different the quality of responses were, not just the speed. Kind of makes me ignore everything else you wrote too, because why would that be correct when you surely haven't validated that before writing it, and y…

What a snide and insulting comment - and plainly wrong.

I literally stated 'I noticed that' - implying I'm using the model.

I'm 'running the model' literally as I write this, I use it every day.

What I was 'wrong' about was the very fine point that '5.3 Codex Spark' is a different model that '5.3 Codex' which is rather a fine point.

I 'thought that I noticed something, but dismissed it' because I value the facts generally more than my intuition. I just so happened that I had that one fact wrong - 'Spark' is technically a different model, so it's not just 'a faster model', it will 'behave differently' , which lends credence to the individual I was responding to.

Re: Gemini 3.1 Pro

#918

Post by replwoacause »

scrollop wrote: post #821
sigmoid10 wrote: post #594

Earlier quoted context omitted.

For me it's Opus 4.6 for researching code/digging through repos, gpt 5.3 codex for writing code, gemini for single hardcore science/math algorithms and grok for things the others refuse to answer or skirt around (e.g. some security/exploitability related queries). Get yourself one of those wrappers that support all models and forget thinking about who has the best model. The question is who has the best model for you…

Using simtheory.ai which is very good, you can switch models within a conversation and use mcps

Are you associated with this somehow?

Re: Gemini 3.1 Pro

#919

Post by girvo »

embedding-shape wrote: post #897
girvo wrote: post #540

Earlier quoted context omitted.

The depressing truth is most I know just run all these tools in /yolo mode or equivalents. Because your coworkers definitely are, and we're stack ranked, so it's a race (literally) to the bottom. Just send it... (All this actually seems to do is push the burden on to their coworkers as reviewers, for what it's worth)

You're mixing up two things though. One is what the agent does "locally", wherever that might be (for me it's inside a VM), and second is what code you actually share or as you call "send". Just because you don't want to gate every change in #1, doesn't mean you're just throwing shit via #2, I'm still reviewing my code as much as before, if not more now, before I consider it ready to be reviewed by others. But I'm se…

> Just because you don't want to gate every change in #1, doesn't mean you're just throwing shit via #2,

Right but in practice from what I've seen at work, it does.

You're right: it shouldn't inherently, but that's what I've been seeing.

> But I'm seemingly also one of the few developers who seem to take responsibility of the code I produce, even if AI happens to have coded it.

Pretty much what I'm getting at, yeah

Re: Gemini 3.1 Pro

#920

Post by surajrmal »

SXX wrote: post #770
surajrmal wrote: post #729

Earlier quoted context omitted.

I think you are underestimating how much cheaper it is for Google to run the workloads compared to competitors. The hardware advantage is real.

Enshittification will begin eventually. Google already cut free limits on AI studio from 100 rpd to 10 rpd so they started cost savings already.

What does that have to do with what I said? Everyone knows that the companies are operating at a loss right now to capture market share in the hope that it's sticky. Google is losing far less money and will not need to get nearly as extreme with how they try to extra money from the product. That honestly makes me feel better about it's long term prospects. And who knows, maybe local llms will prevent it from getting truly bad anyways. Competition tends to keep product quality high.
Post reply on HN