A postmortem of three recent issues • Hacker News Board

Re: A postmortem of three recent issues

#21

Post by bravetraveler » Wed, Sep 17, 2025, 10:10 PM UTC

levocardia wrote: post #16

bravetraveler wrote: post #9

> We don't typically share this level of technical detail about our infrastructure, but the scope and complexity of these issues justified a more comprehensive explanation. Layered in aggrandizing. You host a service, people give you money.

No, what that statement means is "we know that if we just say 'we weren't downgrading performance to save money', you won't believe us, so here is a deep dive on the actual reason it happened"

They can still do the deep dive, that is absolutely convincing. They likely did: distracted before I could finish [work, unfortunately - incident of our own]

My criticism is it's 'puffy'. The 'scope and complexity' for a public postmortem is 'customer-facing'. Otherwise it's a tree/forest scenario.

One might say 'the lady doth protest too much'; this should be routine. It is, elsewhere: see Cloud, Web Hosting, PBX. Pick your decade.

Re: A postmortem of three recent issues

#22

Post by flutas » Wed, Sep 17, 2025, 10:12 PM UTC

extr wrote: post #12

deepdarkforest wrote: post #7

Wow. Sneaky. They do not even state the rate of impact for the XLA bug afaik, which affected everyone, not just claude code users, very vague. Interesting. Claude code made almost half a billion so far[1] (>500m in ARR and its like 9 months old) , and 30% of all users have been impacted at least once, just from the first routing bug. Scary stuff. Their post mortem is basically "evaluations are hard, we relied on vibe…

Is your contention that paying for a service entitles you to zero bugs, ever?

If you paid for a streaming service and the HD option only worked for a random subset of users, and not you, would you complain?

It's a material difference in the product, not just "a bug."

Re: A postmortem of three recent issues

#23

Post by flutas » Wed, Sep 17, 2025, 10:14 PM UTC

extr wrote: post #18

ispeaknumbers wrote: post #11

Earlier quoted context omitted.

I'm not sure if you can claim these were "less prevalent than anecdotal online reports". From their article: > Approximately 30% of Claude Code users had at least one message routed to the wrong server type, resulting in degraded responses. > However, some users were affected more severely, as our routing is "sticky". This meant that once a request was served by the incorrect server, subsequent follow-ups were likely…

I don't know about you but my feed is filled with people claiming that they are surely quantizating the model, Anthropic is purposefully degrading things to save money, etc etc. 70% of users were not impacted. 30% had at least one message degraded. One message is basically nothing. I would have appreciated if they had released the full distribution of impact though.

That 30% is of ALL users, not users who made a request, important to note the weasel wording there.

How many users forget they have a sub? How many get a sub through work and don't use it often?

I'd bet a large number tbh based on other subscription services.

Re: A postmortem of three recent issues

#24

Post by thousand_nights » Wed, Sep 17, 2025, 10:16 PM UTC

extr wrote: post #8

> Incorrect routing affected less than 0.0004% of requests on Google Cloud's Vertex AI between August 27 and September 16. Matches my experience. I use CC through our enterprise Vertex AI account and never noticed any degradation. In general it seems like these bugs, while serious, were substantially less prevalent than anecdotal online reports would have you believe. We are really talking about a ~1-2 week window he…

i don't trust companies anymore because every time there's a worldwide outage they use softspeak like "we're observing elevated amounts of errors for a small subset of users", hours after some CTO approves to change the status page

imho there's a big market gap for companies that are truly honest with customers instead of corporate gaslighting

Re: A postmortem of three recent issues

#25

Post by cyanf » Wed, Sep 17, 2025, 10:17 PM UTC

> On August 29, a routine load balancing change unintentionally increased the number of short-context requests routed to the 1M context servers. At the worst impacted hour on August 31, 16% of Sonnet 4 requests were affected.

Interesting, this implies that the 1M context servers performs worst at low context. Perhaps this is due to some KV cache compression, eviction or sparse attention scheme being applied on these 1M context servers?

Re: A postmortem of three recent issues

#26

Post by jjmarr » Wed, Sep 17, 2025, 10:19 PM UTC

Centigonal wrote: post #19

Wowfunhappy wrote: post #13

> On August 25, we deployed a misconfiguration to the Claude API TPU servers that caused an error during token generation. An issue caused by a runtime performance optimization occasionally assigned a high probability to tokens that should rarely be produced given the context, for example producing Thai or Chinese characters in response to English prompts, or producing obvious syntax errors in code. A small subset of…

LLMs produce a probability distribution for what the next token might be. How you pick the actual word that is printed next from that probability distribution is by using a sampling approach[1]. If your sampling approach is "select the next word randomly from among the top 4 possibilities" and you flip a > sign, you could end up with the behavior described in the OP. [1] Here is an example of two common approaches: h…

The next word can also be selected with weighted randomization and "temperature" is used to control how much weight lower probability tokens get.

I've honestly received the best results in creative writing by ignoring top_k/top_p and simply tuning temperature. Restricting my output to only common words causes everything to feel generic. But Deepseek constantly breaks into Chinese/gibberish/ZALGO! when I go to 1.14.

This isn't related to the "recent issues" but I feel like it's useful advice for anyone trying out AI story creation.

Re: A postmortem of three recent issues

#27

Post by smca » Wed, Sep 17, 2025, 10:20 PM UTC

flutas wrote: post #23

extr wrote: post #18

Earlier quoted context omitted.

I don't know about you but my feed is filled with people claiming that they are surely quantizating the model, Anthropic is purposefully degrading things to save money, etc etc. 70% of users were not impacted. 30% had at least one message degraded. One message is basically nothing. I would have appreciated if they had released the full distribution of impact though.

That 30% is of ALL users, not users who made a request, important to note the weasel wording there. How many users forget they have a sub? How many get a sub through work and don't use it often? I'd bet a large number tbh based on other subscription services.

(I work at Anthropic) It's 30% of all CC users that made a request during that period. We've updated the post to be clearer.

Re: A postmortem of three recent issues

#28

Post by behnamoh » Wed, Sep 17, 2025, 10:20 PM UTC

Reminder that Anthropic is the only AI company that has never released any open-source/weight models.

Re: A postmortem of three recent issues

#29

Post by extr » Wed, Sep 17, 2025, 10:22 PM UTC

flutas wrote: post #23

extr wrote: post #18

Earlier quoted context omitted.

I don't know about you but my feed is filled with people claiming that they are surely quantizating the model, Anthropic is purposefully degrading things to save money, etc etc. 70% of users were not impacted. 30% had at least one message degraded. One message is basically nothing. I would have appreciated if they had released the full distribution of impact though.

That 30% is of ALL users, not users who made a request, important to note the weasel wording there. How many users forget they have a sub? How many get a sub through work and don't use it often? I'd bet a large number tbh based on other subscription services.

That's a pretty cynical read. My personal impression is that Anthropic has a high level of integrity as an organization. Believe what you want, I'm inclined to give them the benefit of the doubt here and move on.

Re: A postmortem of three recent issues

#30

Post by dylan604 » Wed, Sep 17, 2025, 10:22 PM UTC

flutas wrote: post #22

extr wrote: post #12

Earlier quoted context omitted.

Is your contention that paying for a service entitles you to zero bugs, ever?

If you paid for a streaming service and the HD option only worked for a random subset of users, and not you, would you complain? It's a material difference in the product, not just "a bug."

I'd honestly blame my ISP for traffic shaping my connection as a first assumption, and not immediately blame the streaming platform.