Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 3.0 spotted in the wild through A/B testing

ricklamers.io

Post reply on HN
Print view View original

221–230 of 280 posts

Re: Gemini 3.0 spotted in the wild through A/B testing

#221

Post by deaux »

kridsdale3 wrote: post #70
behnamoh wrote: post #20

Earlier quoted context omitted.

Gemini was good when the thinking tokens were shown to the user. As soon as Google replaced those with some thought summary, I stopped finding it as useful. Previously, the thoughts were so organized that I would often read those instead of the final answer.

In the API, the thinking tokens are just a different stream. You can still read them.

They aren't, in the API too they're a thought summary, not nearly as useful as actually showing the thoughts. Anthropic still does provide raw thoughts to this day, showing how it's not necessary to keep a moat. Google and OpenAI don't.

[0] https://ai.google.dev/gemini-api/docs/thinking#summaries

[1] https://discuss.ai.google.dev/t/thoughts-are-missing-cot-not...

[2] https://discuss.ai.google.dev/t/massive-regression-detailed-...

[2]

Re: Gemini 3.0 spotted in the wild through A/B testing

#222

Post by deaux »

whatever1 wrote: post #31
jmkni wrote: post #3

I might be in the minority here but I've consistently found Gemini to be better than ChatGPT, Claude and Deepseek (I get access to all of the pro models through work) Maybe it's just the kind of work I'm doing, a lot of web development with html/scss, and Google has crawled the internet so they have more data to work with. I reckon different models are better at different kinds of work, but Gemini is pretty excellent…

Looking at the responses. How the F have people so wildly different opinions on the relative performance of the same systems?

LLMs: unlimited use cases, all with different performances per model and approach, where a high performance on use case A doesn't mean high performance on use case B. And high performance using approach X for use case A doesn't mean high performance using approach Y for that same use case.

The use case one is bigger than the approach one, but both play a role. Most people only use LLMs for a very specific set of tasks using the same approach every time, so they base their view of them on solely the performance on this task.

That explains all of it.

Re: Gemini 3.0 spotted in the wild through A/B testing

#223

Post by oscaracso »

jjmarr wrote: post #135
gnulinux wrote: post #9

Earlier quoted context omitted.

I agree with you, I consistently find Gemini 2.5 Pro better than Claude and GPT-5 for the following cases: * Creative writing: Gemini is the unmatched winner here by a huge margin. I would personally go so far as to say Gemini 2.5 Pro is the only borderline kinda-sorta usable model for creative writing if you squint your eyes. I use it to criticize my creative writing (poetry, short stories) and no other model unders…

The best model for creative writing is still Deepseek because I can tune temperature to the edge of gibberish for better raw material as that gives me bizarre words. Most models use top_k or top_p or I can't use the full temperature range to promote truly creative word choices. e.g. I asked it to reply to your comment: Oh magnificent, another soul quantifying the relative merits of these digital gods while I languish…

I'm DM'ing for a LessWrong polycule this weekend and you just saved my ass

Re: Gemini 3.0 spotted in the wild through A/B testing

#224

Post by fennecbutt »

montebicyclelo wrote: post #30
jmkni wrote: post #3

I might be in the minority here but I've consistently found Gemini to be better than ChatGPT, Claude and Deepseek (I get access to all of the pro models through work) Maybe it's just the kind of work I'm doing, a lot of web development with html/scss, and Google has crawled the internet so they have more data to work with. I reckon different models are better at different kinds of work, but Gemini is pretty excellent…

Agreed, and its larger context window is fantastic. My workflow: - Convert the whole codebase into a string - Paste it into Gemini - Ask a question People seem to be very taken with "agentic" approaches were the model selects a few files to look at, but I've found it very effective and convenient just to give the model the whole codebase, and then have a conversation with it, get it to output code, modify a file, etc…

Idk though, I've seen many issues occur because of a longer context though. I mean it makes sense, given there are only so many attention heads, the longer the context the less chance attention will pick relevant tokens.

Re: Gemini 3.0 spotted in the wild through A/B testing

#225

Post by blauditore »

That doesn't really look like an actual XBox controller. Yes, it's impressive what it can generate, but not really on par with what professional humans could do.

As usual, the model can get like 95% close to the gold standard, but the last few percent are the hardest ones. I honestly think that most dream scenarios of AI applications will remain dreams for exactly that reason, and the AI bubble will burst badly.

Yes, there are real use cases for the current generation of LLMs and generative models, but they make up only a small fraction of what some of the big companies would like to believe.

Re: Gemini 3.0 spotted in the wild through A/B testing

#226

Post by roryirvine »

gs17 wrote: post #162
dieortin wrote: post #148

Earlier quoted context omitted.

That does not match my experience at all. Basically any Gemini query will run a search.

Which interface are you using for it? I use the gemini.google.com one and most of the time instead of searching it at most pretends to search and hallucinates the result.

Are you telling it to cite sources? I find that doing that helps a lot - sometimes it presents the same result multiple times with a slightly different summary each time, but at least it doesn't outright invent stuff.

Re: Gemini 3.0 spotted in the wild through A/B testing

#227

Post by rjmunro »

kridsdale3 wrote: post #68
Galanwe wrote: post #54

Earlier quoted context omitted.

I usually do that in a 2 step process. Instead of giving the full source code to the model, I will ask it to write a comprehensive, detailed, description of the architecture, intent, and details (including filenames) of the codebase to a Markdown file. Then for each subsequent conversation I would ask the model to use this file as reference. The overall idea is the same, but going through an intermediate file allows…

My special hack on top of what you suggested: Ask it to draw the whole codebase in graphviz compatible graphing markup language. There are various tools out there to render this as an SVG or whatever, to get an actual map of the system. Very helpful when diving in to a big new area.

You can use mermaid format instead of graphviz, then paste it into a markdown file and github will render it inline.

Re: Gemini 3.0 spotted in the wild through A/B testing

#228

Post by bertylicious »

Lucasoato wrote: post #213
elcomet wrote: post #208

I don't understand all the hype for generating SVG with LLM. The task is not really useful, doesn't seem that interesting in single shot as it's really hard, and no human could do it (it would be more useful if the model has visual feedback and could correct the result). And also, since it becomes a popular task, companies will add the examples in their training set, so you're just benchmarking who has the better tex…

One of my co-founders lost the SVG of our startup logo, and the designer who helped us was away on vacation. I really wanted to experiment with some logo animations for an upcoming demo, so I decided to take matters into my own hands. I grabbed a high-quality PNG, gave it to ChatGPT, and managed to recreate the SVG from the image, after quite a bit of prompting and tweaking. But it worked out great!

But isn't this something Inkscape can do since forever?

Re: Gemini 3.0 spotted in the wild through A/B testing

#229

Post by afro88 »

jacquesm wrote: post #216
simonw wrote: post #120

Earlier quoted context omitted.

Here's my collection from the past year. It's definitely better than any of these! https://simonwillison.net/tags/pelican-riding-a-bicycle/

Ok, so we're in the dancing pig stage now. We appreciate that the pig can dance, not how well it dances.

It's quite literally the opposite. Simon is tracking how well the "pig" dances as each model gets better (or worse) at it

Re: Gemini 3.0 spotted in the wild through A/B testing

#230

Post by Topfi »

elcomet wrote: post #208

I don't understand all the hype for generating SVG with LLM. The task is not really useful, doesn't seem that interesting in single shot as it's really hard, and no human could do it (it would be more useful if the model has visual feedback and could correct the result). And also, since it becomes a popular task, companies will add the examples in their training set, so you're just benchmarking who has the better tex…

It goes back to Sparks of AGI [0] unless I am mistaken. Can recommend the talk, one that has stayed in the back of my mind since I first saw it two years ago. Personally, still have major reservations about throwing claims of intelligence or understanding around, but I do agree that SVG code generation can be a very effective source to get a quick and easy to present understanding of a models ability to output code with a rather open ended prompt that needs a high degree of coherence and were a lot of layers depend/build on each other.

Helps that these are eye catching (literally as the output is visual) and easy to grasp. Same reason a lot of hype is created around the web desktops.

[0] https://youtu.be/qbIk7-JPB2c?si=_TNRrxN-_5FOlfy5&t=1342

Post reply on HN