Elevated errors across many models

Post by giancarlostoro » Mon, Dec 15, 2025, 12:59 AM UTC

l1n wrote: post #96

palcu wrote: post #58

Hello, I'm one of the engineers who worked on the incident. We have mitigated the incident as of 14:43 PT / 22:43 UTC. Sorry for the trouble.

Also an engineer on this incident. This was a network routing misconfiguration - an overlapping route advertisement caused traffic to some of our inference backends to be blackholed. Detection took longer than we’d like (about 75 minutes from impact to identification), and some of our normal mitigation paths didn’t work as expected during the incident. The bad route has been removed and service is restored. We’re doi…

Post by awesome_dude » Mon, Dec 15, 2025, 1:05 AM UTC

koakuma-chan wrote: post #104

palcu wrote: post #80

Earlier quoted context omitted.

Thank you! Opening an incident as soon as user impact begins is one of those instincts you develop after handling major incidents for years as an SRE at Google, and now at Anthropic. I was also fortunate to be using Claude at that exact moment (for personal reasons), which meant I could immediately see the severity of the outage.

It's important for companies to use their own products.

Post by dpkirchner » Mon, Dec 15, 2025, 1:20 AM UTC

Buttons840 wrote: post #114

dinkleberg wrote: post #18

Props to them for actually updating their status page as issues are happening rather than hours later. I was working with claude code and hit an API error, checked the status page and sure enough there was an outage. This should be a given for any service that others rely on, but sadly this is seldom the case.

"There's a problem and we already know about it" is so much better than "there's a problem and we don't know about it and/or are hoping it will magically go away and that we won't be embarrassed".

Post by smcleod » Mon, Dec 15, 2025, 1:22 AM UTC

dinkleberg wrote: post #18

Props to them for actually updating their status page as issues are happening rather than hours later. I was working with claude code and hit an API error, checked the status page and sure enough there was an outage. This should be a given for any service that others rely on, but sadly this is seldom the case.

Post by PunchyHamster » Mon, Dec 15, 2025, 1:26 AM UTC

teaearlgraycold wrote: post #27

irishcoffee wrote: post #14

I’m imagining a steampunk dystopia in 50 years: “all world production stopped, LLM hosting went down. The market is in free-fall. Sam, are you there?” Man that cracks me up.

The nice thing is unlike Cloudflare or AWS you can actually host good LLMs locally. I see a future where a non-trivial percentage of devs have an expensive workstation that runs all of the AI locally.