I think from last few releases of these models from all companies, I have not observed much improvements in the response of these models. Their claims and launches are a little over hyped.
Gemini 3
301–310 of 1001 posts
Re: Gemini 3
#302Earlier quoted context omitted.
Google actually changed it somewhat recently (3 months ago, give or take) and you can use Gemini CLI with the "regular" Google AI Pro subscription (~22eur/month). Before that, it required a separate subscription I can't find the announcement anymore, but you can see it under benefits here https://support.google.com/googleone/answer/14534406?hl=en The initial separate subscriptions were confusing at best. Current situ…
I see -- but does this allow me to us the models within "Antigravity" with the same subscription? I poked around and couldn't figure this out.
Might not be decided yet. The AG pricing page says:
"Public preview Individual plan $0/month"
"Coming soon Team plan"
Re: Gemini 3
#303Understanding precisely why Gemini 3 isn't front of the pack on SWE Bench is really what I was hoping to understand here. Especially for a blog post targeted at software developers...
SWEBench-Verified is probably benchmaxxed at this stage. Claude isn't even the top performer, that honor goes to Doubao [1]. Also, the confidence interval for a such a small dataset is about 3 percent points, so these differences could just be up to chance. [1] https://www.swebench.com/
claude 4.5 gets 82% on their own highly customized scaffolding. (parallel compute with a scoring function). That beats Doubao
Re: Gemini 3
#304Earlier quoted context omitted.
Google actually changed it somewhat recently (3 months ago, give or take) and you can use Gemini CLI with the "regular" Google AI Pro subscription (~22eur/month). Before that, it required a separate subscription I can't find the announcement anymore, but you can see it under benefits here https://support.google.com/googleone/answer/14534406?hl=en The initial separate subscriptions were confusing at best. Current situ…
I see -- but does this allow me to us the models within "Antigravity" with the same subscription? I poked around and couldn't figure this out.
I don't know either tbh. I wouldn't be surprised it the answer is no (and it will come later or something like that)
I also tried to use Gemini 3 in my Gemini CLI and it's not available yet (it's available to all Ultra, but not all Pro subscribers), I needed to sign up to a waitlist
All in all, Google is terrible at launching things like that in a concise and understandable way
Re: Gemini 3
#305I am personally impressed by the continued improvement in ARC-AGI-2, where Gemini 3 got 31.1% (vs ChatGPT 5.1's 17.6%). To me this is the kind of problem that does not lend itself well to LLMs - many of the puzzles test the kind of thing that humans intuit because of millions of years of evolution, but these concepts do not necessarily appear in written form (or when they do, it's not clear how they connect to specif…
What I would do if I was in the position of a large company in this space is to arrange an internal team to create an ARC replica, covering very similar puzzles and use that as part of the training.
Ultimately, most benchmarks can be gamed and their real utility is thus short-lived.
But I think this is also fair to use any means to beat it.
Re: Gemini 3
#306I've been so happy to see Google wake up. Many can point to a long history of killed products and soured opinions but you can't deny theyve been the great balancing force (often for good) in the industry. - Gmail vs Outlook - Drive vs Word - Android vs iOS - Worklife balance and high pay vs the low salary grind of before. Theyve done heaps for the industry. Im glad to see signs of life. Particularly in their P/E whic…
All those examples date back to the 2000s. Android has seen some significant improvements, but everything else has stagnated if not enshittified- remember when google told us not to ever worry about deleting anything?- and then started backing up my photos without me asking and are now constantly nagging me to pay them a monthly fee?
They have done a lot, but most of it was in the "don't be evil" days and they are a fading memory.
Re: Gemini 3
#307Earlier quoted context omitted.
He's not a central figure in the narrative, he's a background character. Things he created (MIRI, CFAR, LessWrong) are important to the narrative, the founder isn't. If I had to condense the article, I'd probably cut him out too. Summarization is inherently lossy.
> Eliezer Yudkowsky is a central figure in the article, mentioned multiple times as the intellectual originator of the community from which the "Zizians" splintered. His ideas and organizations are foundational to the entire narrative.
And yet you could eliminate him entirely and the story is still coherent.
The story isn't about Yudkowsky. At each level of summarization you have to make hard decisions about what to keep. Not every story about the United States needs to mention George Washington.
Re: Gemini 3
#308My favorite benchmark is to analyze a very long audio file recording of a management meeting and produce very good notes along with a transcript labeling all the speakers. 2.5 was decently good at generating the summary, but it was terrible at labeling speakers. 3.0 has so far absolutely nailed speaker labeling.
I'd do the transcript and the summary parts separately. Dedicated audio models from vendors like ElevenLabs or Soniox use speaker detection models to produce an accurate speaker based transcript while I'm not necessarily sure that Google's models do so, maybe they just hallucinate the speakers instead.
Re: Gemini 3
#309Every big new model release we see benchmarks like ARC and Humanity's Last Exam climbing higher and higher. My question is, how do we know that these benchmarks are not a part of the training set used for these models? It could easily have been trained to memorize the answers. Even if the datasets haven't been copy pasted directly, I'm sure it has leaked onto the internet to some extent. But I am looking forward to t…
Even if the benchmark themselves are kept secret, the process to create them is not that difficult and anyone with a small team of engineers could make a replica in their own labs to train their models on.
Given the nature of how those models work, you don't need exact replicas.
Re: Gemini 3
#310It’s over for Anthropic. That’s why Google’s cool with Claude being on Azure.
Also probably over for OpenAI