121–130 of 372 posts
Simply search user prompts for curse words and then measure hostility sentiment. User hostility rises as agents fail to meet expectations.
Maybe im overlooking something obvious but how do you 'simply' scan the content of Claude users their prompts?
I am using API mode, and it's clear that there are times when the Claude model just gives up. And it is very noticeable because the model just does the most dumb things possible. "You have a bug in line 23." "Oh yes, this solution is bugged, let me delete the whole feature." That one-line fix I could make even with ChatGPT 3.5 can't just happen. Workflows that I use and are very reproducible start to flake and then f…
Robbing Peter to pay Paul. They are probably resource-constrained, and have determined that it's better to supply a worse answer to more people than to supply a good answer to some while refusing others. Especially knowing that most people probably don't need the best answer 100% of the time.
More: probably don't know if they've got a good answer 100% of the time.
It is interesting to note that this trickery is workable only where the best answers are sufficiently poor. Imagine they ran almost any other kind of online service such email, stock prices or internet banking. Occasionally delivering only half the emails would trigger a customer exodus. But if normal service lost a quarter of emails, they'd have only customers who'd likely never notice half missing.
Why I do not believe this shows Anthropic serves folks a worse model: 1. The percentage drop is too low and oscillating, it goes up and down. 2. The baseline of Sonnet 4.5 (the obvious choice for when they have GPU busy for the next training) should be established to see Opus at some point goes Sonnet level. This was not done but likely we would see a much sharp decline in certain days / periods. The graph would look…
I believe the science, but I've been using it daily and it's been getting worse, noticeably.
If this measure were hardened up a little, it would be really useful.
It feels like an analogue to an employee’s performance over time - you could see in the graphs when Claude is “sick” or “hungover”, when Claude picks up a new side hustle and starts completely phoning it in, or when it’s gunning for a promotion and trying extra hard (significant parameter changes). Pretty neat.
Obviously the anthropomorphising is not real, but it is cool to think of the model’s performance as being a fluid thing you have to work with, and that can be measured like this.
I’m sure some people, most, would prefer that the model’s performance were fixed over time. But come on, this is way more fun.
Earlier quoted context omitted.
but degradation from servers being overloaded would be the type of degradation this SHOULD measure no? Unless it's only intended for measuring their quietly distilling models (which they claim not to do? idk for certain)
Load just makes LLMs behave less deterministically and likely degrade. See: https://thinkingmachines.ai/blog/defeating-nondeterminism-in... They don't have to be malicious operators in this case. It just happens.
Why is this happening?
Earlier quoted context omitted.
noob question: why would increased demand result in decreased intelligence?
An operator at load capacity can either refuse requests, or move the knobs (quantization, thinking time) so requests process faster. Both of those things make customers unhappy, but only one is obvious.
Wouldn't be surprised if they slowly start quantizing their models over time. Makes it easier to scale and reduce operational cost. Also makes a new release have more impact as it will be more notably "better" than what you've been using the past couple of days/weeks.
Earlier quoted context omitted.
Per Anthropic’s RCA linked in Ops post for September 2025 issues: “… To state it plainly: We never reduce model quality due to demand, time of day, or server load. …” So according to Anthropic they are not tweaking quality setting due to demand.
And according to Google, they always delete data if requested. And according to Meta, they always give you ALL the data they have on you when requested.
However, the request form is on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard'.
Earlier quoted context omitted.
This is intentional? I think delivering lower quality than what was advertised and benchmarked is borderline fraud, but YMMV.
Per Anthropic’s RCA linked in Ops post for September 2025 issues: “… To state it plainly: We never reduce model quality due to demand, time of day, or server load. …” So according to Anthropic they are not tweaking quality setting due to demand.