361–370 of 609 posts
Watch out these model are hallucinating lot more https://artificialanalysis.ai/evaluations/omniscience?omnisc...
Edit: Huh... It does score highest in "Omniscience", but also very high in Hallucination Rate (where higher score is worse)...
Watch out these model are hallucinating lot more https://artificialanalysis.ai/evaluations/omniscience?omnisc...
Isn't it the opposite? From the link: Scores range from -100 to 100, where 0 means as many correct as incorrect answers, and negative scores mean more incorrect than correct. Gemini 3 Flash scored +13 in the test, more correct answers than incorrect.
Earlier quoted context omitted.
I'm a significant genAI skeptic. I periodically ask them questions about topics that are subtle or tricky, and somewhat niche, that I know a lot about, and find that they frequently provide extremely bad answers. There have been improvements on some topics, but there's one benchmark question that I have that just about every model I've tried has completely gotten wrong. Tried it on LMArena recently, got a comparison…
So this is an interesting benchmark, because if the answer is actually in the top 3 google results, then my python script that runs a google search, scrapes the top n results and shoves them into a crappy LLM would pass your benchmark too! Which also implies that (for most tasks), most of the weights in a LLM are unnecessary, since they are spent on memorizing the long tail of Common Crawl... but maybe memorizing inf…
Don’t let the “flash” name fool you, this is an amazing model. I have been playing with it for the past few weeks, it’s genuinely my new favorite; it’s so fast and it has such a vast world knowledge that it’s more performant than Claude Opus 4.5 or GPT 5.2 extra high, for a fraction (basically order of magnitude less!!) of the inference time and price
> Don’t let the “flash” name fool you I think it's bad naming on google's part. "flash" implies low quality, fast but not good enough. I get less negative feeling looking at "mini" models.
Earlier quoted context omitted.
Don't the models typically train on their input too? I.e. submitting the question also carries a risk/chance of it getting picked up? I guess they get such a large input of queries that they can only realistically check and therefore use a small fraction? Though maybe they've come up with some clever trick to make use of it anyway?
Given they asked in on LMArena, yes.
I'll need to find a new one, or actually put together a set of questions to use instead of just a single benchmark.
Does anyone else understand what the difference is between Gemini 3 'Thinking' and 'Pro'? Thinking "Solves complex problems" and Pro "Thinks longer for advanced math & code". I assume that these are just different reasoning levels for Gemini 3, but I can't even find mention of there being 2 versions anywhere, and the API doesn't even mention the Thinking-Pro dichotomy.
Really stupid question: How is Gemini-like 'thinking' separate from artificial general intelligence (AGI)? When I ask Gemini 3 Flash this question, the answer is vague but agency comes up a lot. Gemini thinking is always triggered by a query. This seems like a higher-level programming issue to me. Turn it into a loop. Keep the context. Those two things make it costly for sure. But does it make it an AGI? Surely Googl…
Earlier quoted context omitted.
I don't think tricky niche knowledge is the sweet spot for genai and it likely won't be for some time. Instead, it's a great replacement for rote tasks where a less than perfect performance is good enough. Transcription, ocr, boilerplate code generation, etc.
The thing is, I see people use it for tricky niche knowledge all the time; using it as an alternative to doing a Google search. So I want to have a general idea of how good it is at this. I found something that was niche, but not super niche; I could easily find a good, human written answer in the top couple of results of a Google search. But until now, all LLM answers I've gotten for it have been complete hallucinat…
Earlier quoted context omitted.
OpenAI made a huge mistake neglecting fast inferencing models. Their strategy was gpt 5 for everything, which hasn't worked out at all. I'm really not sure what model OpenAI wants me to use for my applications that require lower latency. If I follow their advice in their API docs about which models I should use for faster responses I get told either use GPT 5 low thinking, or replace gpt 5 with gpt 4.1, or switch to…
Hard to find info but I think the -chat versions of 5.1 and 5.2 (gpt-5.2-chat) are what you're looking for. They might just be an alias for the same model with very low reasoning though. I've seen other providers do the same thing, where they offer a reasoning and non reasoning endpoint. Seems to work well enough.
Earlier quoted context omitted.
I’ve fully switched over to Gemini now. It seems significantly more useful, and is less of an automatic glaze machine that just restates your question and how smart you are for asking it.
How do I get Gemini to be more proactive in finding/double-checking itself against new world information and doing searches? For that reason I still find chatgpt way better for me, many things I ask it first goes off to do online research and has up to date information - which is surprising as you would expect Google to be way better at this. For example, was asking Gemini 3 Pro recently about how to do something wit…
That epistemic calibration is is something they are capable of thinking through if you point it out. But they aren’t trained to stop and ask/check themselves on how confident do they have a right to be. This is a meta cognitive interrupt that is socialized into girls between 6 and 9 and is socialized into boys between 11-13. While meta cognitive interrupt to calibrate to appropriate confidence levels of knowledge is a cognitive skill that models aren’t taught and humans learn socially by pissing off other humans. It’s why we get pissed off st models when they correct ua with old bad data. Our anger is the training tool to stop doing that. Just that they can’t take in that training signal at inference time
Earlier quoted context omitted.
The thing is, I see people use it for tricky niche knowledge all the time; using it as an alternative to doing a Google search. So I want to have a general idea of how good it is at this. I found something that was niche, but not super niche; I could easily find a good, human written answer in the top couple of results of a Google search. But until now, all LLM answers I've gotten for it have been complete hallucinat…
And Google themselves obviously believe that too as they happily insert AI summaries at the top of most serps now.