Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 3.0 spotted in the wild through A/B testing

ricklamers.io

Post reply on HN
Print view View original

251–260 of 280 posts

Re: Gemini 3.0 spotted in the wild through A/B testing

#251

Post by computerthings »

AlotOfReading wrote: post #132
altopex wrote: post #127

Earlier quoted context omitted.

LLMs have issues with creative tasks that might not be obvious for light users. Using them for an RPG campaign could work if the bar is low and it's the first couple of times you use it. But after a while, you start to identify repeated patterns and guard rails. The weights of the models are static. It's always predicting what the best association is between the input prompt and whatever tokens its spitting out with…

You're talking about a very different use than the one suggested upthread: I use it to criticize my creative writing (poetry, short stories) and no other model understands nuances as much as Gemini. In that use case, the lack of creativity isn't as severe an issue because the goal is to check if what's being communicated is accessible even to "a person" without strong critical reading skills. All the creativity is st…

[dead]

Re: Gemini 3.0 spotted in the wild through A/B testing

#252

Post by bgwalter »

jstanley wrote: post #250
bgwalter wrote: post #247

Earlier quoted context omitted.

How do we know it's not just a mashup of existing pictures? All generated pelicans on bikes look somewhat cartoonish and use historical or artsy bikes. This is training material from 2015: https://www.behance.net/gallery/29122113/Pelican-on-bikes-wi... There are other such images. Not an image model? How do we know that they don't convert all images to svg and train an LLM on it? How do we know that they do not cheat…

"it's not impressive because they might have cheated" isn't a great argument.

The generated picture is not impressive and the excuse in this subthread was that an svg is created directly without using an image model. I offer alternative explanations why svg creation might not be impressive OR ALTERNATIVELY why they may have faked even a bad result because it is a popular benchmark (faking a perfect result would be too obvious).

But since everything is closed source with any number of potential special case hacks, we won't know.

Re: Gemini 3.0 spotted in the wild through A/B testing

#253

Post by vintermann »

jmkni wrote: post #3

I might be in the minority here but I've consistently found Gemini to be better than ChatGPT, Claude and Deepseek (I get access to all of the pro models through work) Maybe it's just the kind of work I'm doing, a lot of web development with html/scss, and Google has crawled the internet so they have more data to work with. I reckon different models are better at different kinds of work, but Gemini is pretty excellent…

I do very different work, or try to - historical HTR is unfortunately so bad even with the top models that the results aren't useful, but I keep trying new models. But there's a historian on substack (Mark Humphries) who's also trying new models, and he also thinks he's gotten Gemini 3 output in A/B tests. He's very impressed with it:

https://generativehistory.substack.com/p/has-google-quietly-...

I can attest to what he's saying that existing models are especially useless on tabular handwritten data such as ledgers.

Re: Gemini 3.0 spotted in the wild through A/B testing

#254

Post by nprateem »

Gemini has developed an annoying habit of writing blog posts or news articles in response to questions. That and continually blowing smoke up my ass.

When I tell it I don't need its validation it just replies "Yes, you've got me. That is the sharpest comment you could have made", etc etc

Re: Gemini 3.0 spotted in the wild through A/B testing

#255

Post by TurboSkyline »

grej wrote: post #73

My strange observation is that Gemini 2.5 Pro is maybe the best model overall for many use cases, but starting from the first chat. In other words, if it has all the context it needs and produces one output, it's excellent. The longer a chat goes, it gets worse very quickly. Which is strange because it has a much longer context window than other models. I have found a good way to use it is to drop the entire huge con…

Hey, this has been my experience, too! I like Gemini because I’ve told it the tone and style I like my answers in and the first answer is very, very on point with that. But several times I’ve noticed that if I ask follow-up questions, the style immediately changes for the worse, often no longer following my preferences. I’ve also noticed that in follow-ups it makes really bad analogies that are not suitable at all for the kind of audience that the first response is catered to. I’ve been clicking the thumbs-down button every time I’ve seen this and commenting on the change in style and quality, so hopefully the training process will ingest that at some point.

Re: Gemini 3.0 spotted in the wild through A/B testing

#256

Post by Davidzheng »

elcomet wrote: post #208

I don't understand all the hype for generating SVG with LLM. The task is not really useful, doesn't seem that interesting in single shot as it's really hard, and no human could do it (it would be more useful if the model has visual feedback and could correct the result). And also, since it becomes a popular task, companies will add the examples in their training set, so you're just benchmarking who has the better tex…

It's obviously a pointless benchmark-but it's fun so people like doing it

Re: Gemini 3.0 spotted in the wild through A/B testing

#257

Post by behnamoh »

deaux wrote: post #221
kridsdale3 wrote: post #70

Earlier quoted context omitted.

In the API, the thinking tokens are just a different stream. You can still read them.

They aren't, in the API too they're a thought summary, not nearly as useful as actually showing the thoughts. Anthropic still does provide raw thoughts to this day, showing how it's not necessary to keep a moat. Google and OpenAI don't. [0] https://ai.google.dev/gemini-api/docs/thinking#summaries [1] https://discuss.ai.google.dev/t/thoughts-are-missing-cot-not... [2] https://discuss.ai.google.dev/t/massive-regression…

to be fair, Anthropic reasoning models don't have long thinking to begin with, and I find their reasoning pretty useless compared to what Gemini used to do.

Re: Gemini 3.0 spotted in the wild through A/B testing

#258

Post by fkyoureadthedoc »

OtherShrezzing wrote: post #113
jacquesm wrote: post #106

Earlier quoted context omitted.

That's good? Looks like complete crap to me.

I like the pelican riding a bike test, but my standards for what’s “good” seem higher than generally expected by others. The models can generate hyper realistic renders of pelicans riding bikes in png format. They also have perfect knowledge of the SVG spec, and comprehensive knowledge of most human creative artistic endeavours. They should be able to produce astonishing results for the request. I don’t want to see a…

But you're never going to get that out of the prompt that is being used to generate these Pelicans. You're judging it on something that's not even being attempted.

Re: Gemini 3.0 spotted in the wild through A/B testing

#259

Post by sd9 »

suminjs wrote: post #244

While the speed and terseness of models like GPT-5 are great for simple coding tasks or short answers, the verbosity of Gemini is a massive asset for high-stakes tasks where depth matters.

I find verbosity annoying. I prefer depth/accuracy/structure without extra words.

Maybe Gemini still wins on that front anyway.

Re: Gemini 3.0 spotted in the wild through A/B testing

#260

Post by RayVR »

jmkni wrote: post #3

I might be in the minority here but I've consistently found Gemini to be better than ChatGPT, Claude and Deepseek (I get access to all of the pro models through work) Maybe it's just the kind of work I'm doing, a lot of web development with html/scss, and Google has crawled the internet so they have more data to work with. I reckon different models are better at different kinds of work, but Gemini is pretty excellent…

Interesting. Not my experience at all. It makes mistakes that GPT-4 used to make: mixing languages (using Python syntax in C++ when I never asked any Python questions), imagining API calls that don’t exist in Google’s own products, writing 50 lines of C++ then inserting pseudo code or completely broken syntax.
Post reply on HN