Gemini 3.0 spotted in the wild through A/B testing
ricklamers.io
Gemini 3.0 spotted in the wild through A/B testing
1–10 of 280 posts
Re: Gemini 3.0 spotted in the wild through A/B testing
#2Has been ongoing for roughly a month now, with a variety of checkpoints along the usual speculation. As it stands, I'd just wait for the official announcement, prior to making any judgement. What their release plans are, whether a checkpoint is a possible replacement for Pro, Flash, Flash Lite, a new category of model, won't be released at all, etc. we cannot know.
More importantly, because of the way AIStudio does A/B testing, the only output we can get is for a single prompt and I personally maintain that outside of getting some basic understanding on speed, latency and prompt adherence, output from one single prompt is not a good measure for performance in the day-to-day. It also, naturally, cannot tell us a thing about handling multi file ingest and tool calls, but hype will be hype.
That there are people who are ranking alleged performance solely by one-prompt A/B testing output says a lot about how unprofessionally some evaluate model performance.
Not saying the Gemini 3.0 models couldn't be competitive, I just want to caution against getting caught up in over-excitement and possible disappointment. Same reason I dislike speculative content in general, it rarely is put into the proper context cause that isn't as eyecatching.
Re: Gemini 3.0 spotted in the wild through A/B testing
#3I might be in the minority here but I've consistently found Gemini to be better than ChatGPT, Claude and Deepseek (I get access to all of the pro models through work)
Maybe it's just the kind of work I'm doing, a lot of web development with html/scss, and Google has crawled the internet so they have more data to work with.
I reckon different models are better at different kinds of work, but Gemini is pretty excellent at UI/UX web development, in my experience
Very excited to see what 3.0 is like
Re: Gemini 3.0 spotted in the wild through A/B testing
#4I might be in the minority here but I've consistently found Gemini to be better than ChatGPT, Claude and Deepseek (I get access to all of the pro models through work) Maybe it's just the kind of work I'm doing, a lot of web development with html/scss, and Google has crawled the internet so they have more data to work with. I reckon different models are better at different kinds of work, but Gemini is pretty excellent…
What's your use case? We've found Gemini to work well with large context windows, but it sucks at calling MCPs and is worse at writing code
Re: Gemini 3.0 spotted in the wild through A/B testing
#5Rumour is a release on the 22nd I believe
Re: Gemini 3.0 spotted in the wild through A/B testing
#6I might be in the minority here but I've consistently found Gemini to be better than ChatGPT, Claude and Deepseek (I get access to all of the pro models through work) Maybe it's just the kind of work I'm doing, a lot of web development with html/scss, and Google has crawled the internet so they have more data to work with. I reckon different models are better at different kinds of work, but Gemini is pretty excellent…
What's your use case? We've found Gemini to work well with large context windows, but it sucks at calling MCPs and is worse at writing code
Building out user interfaces in html and scss (mainly in Angular)
You need to give it detailed instructions and be willing to do the plumbing yourself, but we've found it to be very good at it
Re: Gemini 3.0 spotted in the wild through A/B testing
#7I might be in the minority here but I've consistently found Gemini to be better than ChatGPT, Claude and Deepseek (I get access to all of the pro models through work) Maybe it's just the kind of work I'm doing, a lot of web development with html/scss, and Google has crawled the internet so they have more data to work with. I reckon different models are better at different kinds of work, but Gemini is pretty excellent…
I use LLMs a lot for health related things (e.g. “Here are 6 bloodwork panels over the past 12 months, here’s a list of medical information, please identify trends/insights/correlations [etc]”)
I default to using ChatGPT since I like the Projects feature (missing from Gemini I think?).
I occasionally run the same prompts in Gemini to compare. A couple notes:
1) Gemini is faster to respond in 100% of cases (most of my prompts kick ChatGPT into thinking mode). ChatGPT is slow.
2) The longer thinking time doesn’t seem to correlate with better quality responses. If anything, Gemini provides better quality analyses despite shorter response time.
3) Gemini (and Claude) are more censored than ChatGPT. Gemini/Claude often refuse medical related prompts, while ChatGPT will answer.
Re: Gemini 3.0 spotted in the wild through A/B testing
#8I might be in the minority here but I've consistently found Gemini to be better than ChatGPT, Claude and Deepseek (I get access to all of the pro models through work) Maybe it's just the kind of work I'm doing, a lot of web development with html/scss, and Google has crawled the internet so they have more data to work with. I reckon different models are better at different kinds of work, but Gemini is pretty excellent…
[deleted]
Re: Gemini 3.0 spotted in the wild through A/B testing
#9I might be in the minority here but I've consistently found Gemini to be better than ChatGPT, Claude and Deepseek (I get access to all of the pro models through work) Maybe it's just the kind of work I'm doing, a lot of web development with html/scss, and Google has crawled the internet so they have more data to work with. I reckon different models are better at different kinds of work, but Gemini is pretty excellent…
I agree with you, I consistently find Gemini 2.5 Pro better than Claude and GPT-5 for the following cases:
* Creative writing: Gemini is the unmatched winner here by a huge margin. I would personally go so far as to say Gemini 2.5 Pro is the only borderline kinda-sorta usable model for creative writing if you squint your eyes. I use it to criticize my creative writing (poetry, short stories) and no other model understands nuances as much as Gemini. Of course, all models are still pretty much terrible at this, especially in writing poetry.
* Complex reasoning (e.g. undergrad/grad level math): Gemini is the best here imho by a tiny margin. Claude Opus 4.1 and Sonnet 4.5 are pretty close but imho Gemini 2.5 writes more predictably correct answers. My bias is algebra stuff, I usually ask things about commutative algebra, linear algebra, category theory, group theory, algebraic geometry, algebraic topology etc.
On the other hand Gemini is significantly worse than Claude and GPT-5 when it comes to agentic behavior, such as searching a huge codebase to answer an open ended question and write a refactor. It seems like its tool calling behavior is buggy and doesn't work consistently in Copilot/Cursor.
Overall, I still think Gemini 2.5 Pro is the smartest overall model, but of course you need to use different models for different tasks.
Re: Gemini 3.0 spotted in the wild through A/B testing
#10I might be in the minority here but I've consistently found Gemini to be better than ChatGPT, Claude and Deepseek (I get access to all of the pro models through work) Maybe it's just the kind of work I'm doing, a lot of web development with html/scss, and Google has crawled the internet so they have more data to work with. I reckon different models are better at different kinds of work, but Gemini is pretty excellent…
I like Gemini 2.5 as a chatbot, but it has been mostly useless as an agent comparing to Claude Code (at least for my complex tasks)