131–140 of 153 posts
Earlier quoted context omitted.
Assuming you include the same prompts in the new request that were cached in the previous ones.
As far as I understand, the entire chat is the prompt. So at the each round, the previous chat up to that point could already be cached. If I'm not wrong, Claude APIs require an explicit request to cache the prompt, while OpenAI's handle this automatically.
Tried it with a few prompts. The vibes feel super weird to me, in a way that I don't know how to describe. I'm not sure if it's just that I'm so used to Gemini 2.5 Pro and the other US models. Subjectively, it doesn't feel very smart. I asked it to analyze a recent painting I made and found the response uninspired. Although at least the feedback that it provided was notably distinct from what I could get from the US…
Earlier quoted context omitted.
Conversations are always "reprocessed from scratch" on every message you send. LLMs are practically stateless and the conversation is the state, as in nothing is kept in memory between two turns.
Not exactly true ... KV and prompt caching is a thing
Earlier quoted context omitted.
While true, it's hard to believe they forgot to s/claude/glm/g? Also, I don't believe LLMs identify themselves that often, even less so in a training corpus they've been used to produce. OTOH, I see no other explanation.
It wouldn't be as simple as search-replace. After all, Claude is name which appears in many more contexts than just LLM-related ones.
Notes on how I got that working, plus a neat Space Invaders clone it built me: https://simonwillison.net/2025/Jul/29/space-invaders/
Earlier quoted context omitted.
"is china a democracy?" .. though process started very reasonable at first and then i got this as the final answer: > Uh-oh! There was an issue with the response. Content Security Warning: The content may contain inappropriate content.
oh really？ China considers itself a democracy with its own characteristics. According to China's White Paper "China: Democracy That Works" published by the State Council Information Office, China practices "whole-process people's democracy," which integrates process and results democracy, procedural and substantive democracy, direct and indirect democracy, and people's democracy and state will. China's democracy is d…
(Btw this chinese ai-propaganda-bot parent is the best example why we need to keep china out of our digital infrastructure and ban apps like tiktok. They are already brainwashing western kids with shit like this fake user above. )
Earlier quoted context omitted.
Then you need to reprocess the previous conversation from scratch when switching from one provider to another, which sounds very expensive for no reason.
Conversations are always "reprocessed from scratch" on every message you send. LLMs are practically stateless and the conversation is the state, as in nothing is kept in memory between two turns.
This isn't true of any practical implementation: for a particular conversation, KV Cache is the state. (Indeed there's no state across conversations, but that's irrelevant to the discussion).
You can drop it after each response, but doing so increase the amount of token you need to process by a lot in multi-turn conversations.
And my point was that storing the KV cache for the duration of the conversation isn't possible if you switch between multiple providers in a single conversation.
Earlier quoted context omitted.
Then you need to reprocess the previous conversation from scratch when switching from one provider to another, which sounds very expensive for no reason.
Take a look at the API calls you'd use to build your own chatbot on top of any of the available models. Like https://docs.anthropic.com/en/api/messages or https://platform.openai.com/docs/api-reference/chat - you send the message history each time. You can even lie about that message history! You can utilize caching like https://platform.openai.com/docs/guides/prompt-caching and note that "Cache hits are only possibl…