Live data from Hacker News

Gemini 3.0 spotted in the wild through A/B testing

ricklamers.io

71–80 of 280 posts

Re: Gemini 3.0 spotted in the wild through A/B testing

#71
post #66

https://x.com/chetaslua is experimenting a lot with Gemini 3 and posting its results (various web desktops, a vampire survivor clone which is actually very playable, voxel 3d models, other game clones, SVG etc). They look really good, specially when they are one-shot.

This was cool: https://codepen.io/ChetasLua/pen/yyezLjN Somewhat amusing 4th wall breaking if you open Python from the terminal in the fake Windows. Examples: 1. If you try to print something using the "Python" print keyword, it opens a print dialog in your browser. 2. If you try to open a file using the "Python" open keyword, it opens a new browser tab trying to access that file. That is, it's forwarding the print a…

Ah, that's because the "python" is actually just using javascript evals.

} else if (mode === 'python') { if (cmd === 'exit()') { mode = 'sh'; } else { try { // Safe(ish) eval for demo purposes. // In production, never use eval. Use a JS parser library. // Mapping JS math to appear somewhat pythonesque let result = eval(cmd); if (result !== undefined) output(String(result)); } catch (e) { output(`Traceback (most recent call last):\n File "", line 1, in \n${e.name}: ${e.message}`, true); } }

Re: Gemini 3.0 spotted in the wild through A/B testing

#72
post #51

> Gemini 3.0 is one of the most anticipated releases in AI at the moment because of the expected advances in coding performance. Based on what I'm hearing from friends who work at Google and are using it for coding, we're all going to be very disappointed. Edit: It sound like they don't actually have Gemini 3 access, which would explain why they aren't happy with it.

Gemini 3.0 isn't broadly available inside Google. There's are "Gemini for Google" fine-tuned versions of 2.5 Pro and 2.5 Flash, but there's been no broad availability of any 3.0 models yet.

Source: I work at Google (on payments, not any AI teams). Opinions mine not Google's.

Re: Gemini 3.0 spotted in the wild through A/B testing

#73
My strange observation is that Gemini 2.5 Pro is maybe the best model overall for many use cases, but starting from the first chat. In other words, if it has all the context it needs and produces one output, it's excellent. The longer a chat goes, it gets worse very quickly. Which is strange because it has a much longer context window than other models. I have found a good way to use it is to drop the entire huge context of a while project (200k-ish tokens) into the chat window and ask one well formed question, then kill the chat.

Re: Gemini 3.0 spotted in the wild through A/B testing

#74
post #51

> Gemini 3.0 is one of the most anticipated releases in AI at the moment because of the expected advances in coding performance. Based on what I'm hearing from friends who work at Google and are using it for coding, we're all going to be very disappointed. Edit: It sound like they don't actually have Gemini 3 access, which would explain why they aren't happy with it.

Hate to spoil this excitement, but we at Google do not have Gemini 3 available to us for use in Vibecoding.

Re: Gemini 3.0 spotted in the wild through A/B testing

#75
post #15

Earlier quoted context omitted.

I find Claude and Gemini to be wildly inferior to ChatGPT when it comes to doing searches to establish grounding. Gemini seems to do a handful of searches and then make shit up, where ChatGPT will do dozens or even hundreds of searches - and do searches based on what it finds in earlier ones.

Try "AI Mode" on Google.com (Disclaimer, I recently joined the team that makes this product). It isn't Gemini (the product, those are different orgs) though there may (deliberately left ambiguous) be overlap in LLM level bytes. My recommendation for you in this use-case comes from the fact that AI Mode is a product that is built to be a good search engine first, presented to you in the interface of an AI Chatbot. Rat…

I have been playing with it recently and, yeah, it's much better than Gemini. It's still seems to be single-shot though - as in, it reads your text, thinks about it for a bit, kicks off searches, reads those searches, thinks, and answers. It never, as far as I can tell, kicks off new searches based on the thinking it did after the initial searches - whereas chatgpt will often do half a dozen or more iterations of that.

Re: Gemini 3.0 spotted in the wild through A/B testing

#76
post #3

I might be in the minority here but I've consistently found Gemini to be better than ChatGPT, Claude and Deepseek (I get access to all of the pro models through work) Maybe it's just the kind of work I'm doing, a lot of web development with html/scss, and Google has crawled the internet so they have more data to work with. I reckon different models are better at different kinds of work, but Gemini is pretty excellent…

Looking at the responses. How the F have people so wildly different opinions on the relative performance of the same systems?

Different prompts/approaches?

I "grew up", as it were, on StackOverflow, when I was in my early dev days and didn't have a clue what I was doing I asked question after question on SO and learned very quickly the difference between asking a good question vs asking a bad one

There is a great Jon Skeet blog post from back in the day called "Writing the perfect question" - https://codeblog.jonskeet.uk/2010/08/29/writing-the-perfect-...

I think this is as valid as ever in the age of AI, you will get much better output from any of these chatbots if you learn and understand how to ask a good question.

Re: Gemini 3.0 spotted in the wild through A/B testing

#77

I hope Gemini 3.0 will also be free, like Gemini 2.5 Pro is if you use the CLI or the right subdomain.

2.5 Pro is limited to 100 request per day every where I think. My Gemini CLI is authed through the Google Account (not API key) and after 100 requests it switches to Flash, API keys are also limited to 100 requests each (and I think there's a limit on free keys now as well)

Re: Gemini 3.0 spotted in the wild through A/B testing

#78
post #15
post #3

I might be in the minority here but I've consistently found Gemini to be better than ChatGPT, Claude and Deepseek (I get access to all of the pro models through work) Maybe it's just the kind of work I'm doing, a lot of web development with html/scss, and Google has crawled the internet so they have more data to work with. I reckon different models are better at different kinds of work, but Gemini is pretty excellent…

I find Claude and Gemini to be wildly inferior to ChatGPT when it comes to doing searches to establish grounding. Gemini seems to do a handful of searches and then make shit up, where ChatGPT will do dozens or even hundreds of searches - and do searches based on what it finds in earlier ones.

That's my experience as well. Gemini doesn't seem interested in doing searches outside of Deep Research mode, which is kind of funny given it should have the easiest access to a top search engine.

Re: Gemini 3.0 spotted in the wild through A/B testing

#79

Earlier quoted context omitted.

> consistently found Gemini to be better than ChatGPT, Claude and Deepseek I used Pro Mode in ChatGPT since it was available, and tried Claude, Gemini, Deepseek and more from time to time, but none of them ever get close to Pro Mode, it's just insanely better than everything. So when I hear people comparing "X to ChatGPT", are you testing against the best ChatGPT has to offer, or are you comparing it to "Auto" and ca…

It seems you also did not compare ChatGPT to the best offers of the competitors, as you did not mention Gemini Deepthink mode which is Google's alternative to GPT's Pro mode.

> It seems you also did not compare ChatGPT to the best offers of the competitors

I am, continuously, and have been since ChatGPT Pro appeared.

Re: Gemini 3.0 spotted in the wild through A/B testing

#80
post #73

My strange observation is that Gemini 2.5 Pro is maybe the best model overall for many use cases, but starting from the first chat. In other words, if it has all the context it needs and produces one output, it's excellent. The longer a chat goes, it gets worse very quickly. Which is strange because it has a much longer context window than other models. I have found a good way to use it is to drop the entire huge con…

> The longer a chat goes, it gets worse very quickly.

This has been the same for every single LLM I've used, ever, they're all terrible at that.

So terrible that I've stopped going beyond two messages in total. If it doesn't get it right at the first try, its more and more unlikely to get it right for every message you add.

Better to always start fresh, iterate on the initial prompt instead.

Post reply on HN