Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 3.1 Pro

blog.google

Post reply on HN
Print view View original

471–480 of 951 posts

Re: Gemini 3.1 Pro

#471

Post by varispeed »

spankalee wrote: post #199

I hope this works better than 3.0 Pro I'm a former Googler and know some people near the team, so I mildly root for them to at least do well, but Gemini is consistently the most frustrating model I've used for development. It's stunningly good at reasoning, design, and generating the raw code, but it just falls over a lot when actually trying to get things done, especially compared to Claude Opus. Within VS Code Copi…

> stuck in loops

I wonder if there is some form of cheating. Many times I found that after a while Gemini becomes like a Markov chain spouting nonsense on repeat suddenly and doesn't react to user input anymore.

Re: Gemini 3.1 Pro

#472

Post by mbh159 »

77.1% on ARC-AGI-2 and still can't stop adding drive-by refactors. ARC-AGI-2 tests novel pattern induction, it's genuinely hard to fake and the improvement is real. But it doesn't measure task scoping, instruction adherence, or knowing when to stop. Those are the capabilities practitioners actually need from a coding agent. We have excellent benchmarks for reasoning. We have almost nothing that measures reliability in agentic loops. That gap explains this thread.

Re: Gemini 3.1 Pro

#473

Post by FergusArgyll »

orbital-decay wrote: post #462
FergusArgyll wrote: post #446

Earlier quoted context omitted.

They hide the CoT because they don't want competitors to train on it

Training on the CoT itself is pretty dubious since it's reward hacked to some degree (as evident from e.g. GLM-4.7 which tried pulling that with 3.0 Pro, and ended up repeating Model Armor injections without really understanding/following them). In any case they aren't trying to hide it particularly hard.

> In any case they aren't trying to hide it particularly hard.

What does that mean? Are you able to read the raw cot? how?

Re: Gemini 3.1 Pro

#474

Post by tasuki »

takoid wrote: post #43
sho_hn wrote: post #11

Earlier quoted context omitted.

The touted SVG improvements make me excited for animated pelicans.

I just gave it a shot and this is what I got: https://codepen.io/takoid/pen/wBWLOKj The model thought for over 5 minutes to produce this. It's not quite photorealistic (some parts are definitely "off"), but this is definitely a significant leap in complexity.

That's a good pelican. What I like the most is that the SVG is nice and readable. If only Inkscape could output nice SVG like this!

Re: Gemini 3.1 Pro

#476

Post by rvnx »

ghurtado wrote: post #448
gnatolf wrote: post #399

Earlier quoted context omitted.

So let's put things we're interested in in the benchmarks. I'm not against pelicans!

I think the reason the pelican example is great is because it's bizarre enough that it's unlikely that to appear in the training as one unified picture. If we picked something more common, like say, a hot dog with toppings, then the training contamination is much harder to control.

It's the most common SVG test, it's the equivalent of Will Smith eating spaghettis, so obviously they benchmax toward it

Re: Gemini 3.1 Pro

#477

Post by casey2 »

jeffybefffy519 wrote: post #440

Someone needs to make an actual good benchmark for LLM's that matches real world expectations, theres more to benchmarks than accuracy against a dataset.

We don't need real world benchmarks, if they were good for real world tasks people would use them We need scientific benchmarks that tease out the nature of intelligence. There are plenty of unsaturated benchmarks. Solving chess using "mostly" language modeling is still an open problem. And beyond that creating a machine that can explain why that move is likely optimal at some depth. AI that can predict the output of another AI.

Re: Gemini 3.1 Pro

#478

Post by small_model »

TZubiri wrote: post #468
small_model wrote: post #444

Earlier quoted context omitted.

I have a codebase (personal project) and every time there is a new Claude Opus model I get it to do a full code review. Never had any breakages in last couple of model updates. Worried one day it just generates a binary and deletes all the code.

No version control?

I was being facetious, I mean one day models might skip the middle man of code and compilation and take your specs and produce an ultra efficent binary.

Re: Gemini 3.1 Pro

#479

Post by oblio »

minimaxir wrote: post #23

Price is unchanged from Gemini 3 Pro: $2/M input, $12/M output. https://ai.google.dev/gemini-api/docs/pricing Knowledge cutoff is unchanged at Jan 2025. Gemini 3.1 Pro supports "medium" thinking where Gemini 3 did not: https://ai.google.dev/gemini-api/docs/gemini-3 Compare to Opus 4.6's $5/M input, $25/M output. If Gemini 3.1 Pro does indeed have similar performance, the price difference is notable.

> Knowledge cutoff is unchanged at Jan 2025.

Isn't that a bit old?

Re: Gemini 3.1 Pro

#480

Post by dist-epoch »

svara wrote: post #456
dist-epoch wrote: post #390

Earlier quoted context omitted.

Musk said Grok 5 is currently being trained, and it has 7 trillion params (Grok 4 had 3)

My understanding is that all recent gains are from post training and no one (publicly) knows how much scaling pretraining will still help at this point. Happy to learn more about this if anyone has more information.

You gain more benefit spending compute on post-training than on pre-training.

But scaling pre-training is still worth it if you can afford it.

Post reply on HN