Claude vs. Gemini: Testing on 1M Tokens of Context

Post by IanCal » Tue, Aug 12, 2025, 7:36 PM UTC

XenophileJKO wrote: post #18

daft_pink wrote: post #12

i’m really curious how well they perform with a long chat history. i find that gemini often gets confused when the context is long enough and starts responding to prior prompts, using the cli or it’s gem chat window.

From my experience. Gemini is REALLY bad about context blending. It can't keep track of what I said and what it said in a conversation under 200K tokens. It blends concepts and statements up, then refers to some fabricated hybrid fact or comment. Gemini has done this in ways that I haven't seen in the recent or current generation models from OpenAI or Anthropic. It really surprised me that Gemini performs so well in…

Post by cma » Tue, Aug 12, 2025, 8:24 PM UTC

HackerThemAll wrote: post #23

What people seem to miss very hard is that they get interactive chat mode of all the models, including the best and newest (Gemini 2.5 Pro, 2.5 Flash, 2.5 Flash Lite and older) totally for free. I mean when working from chat at https://aistudio.google.com/ the entire 1M context window and all is totally free of charge. You really get a very good AI for nothing. https://i.imgur.com/pgfRrZY.png

Post by relatedtitle » Tue, Aug 12, 2025, 8:52 PM UTC

cma wrote: post #24

HackerThemAll wrote: post #23

What people seem to miss very hard is that they get interactive chat mode of all the models, including the best and newest (Gemini 2.5 Pro, 2.5 Flash, 2.5 Flash Lite and older) totally for free. I mean when working from chat at https://aistudio.google.com/ the entire 1M context window and all is totally free of charge. You really get a very good AI for nothing. https://i.imgur.com/pgfRrZY.png

Can you opt out of them training on your data in that free tier?

Post by [deleted] » Tue, Aug 12, 2025, 9:15 PM UTC

HackerThemAll wrote: post #23

What people seem to miss very hard is that they get interactive chat mode of all the models, including the best and newest (Gemini 2.5 Pro, 2.5 Flash, 2.5 Flash Lite and older) totally for free. I mean when working from chat at https://aistudio.google.com/ the entire 1M context window and all is totally free of charge. You really get a very good AI for nothing. https://i.imgur.com/pgfRrZY.png

Post by chrismustcode » Tue, Aug 12, 2025, 9:25 PM UTC

koakuma-chan wrote: post #21

PeterStuer wrote: post #17

Earlier quoted context omitted.

The series is 1,084,170 words. At let's say 1.4 tokens per word, this would not fit, but it is getting close.

It's 2M tokens for Gemini.