Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elevated errors across many models

status.claude.com

Post reply on HN
Print view View original

111–120 of 170 posts

Re: Elevated errors across many models

#111

Post by mattdeboard »

wouldbecouldbe wrote: post #109
l1n wrote: post #96

Earlier quoted context omitted.

Also an engineer on this incident. This was a network routing misconfiguration - an overlapping route advertisement caused traffic to some of our inference backends to be blackholed. Detection took longer than we’d like (about 75 minutes from impact to identification), and some of our normal mitigation paths didn’t work as expected during the incident. The bad route has been removed and service is restored. We’re doi…

Trying to understand what this means. Did the bad route cause an overload? Was there a code error on that route that wasn’t spotted? Was it a code issue or an instance that broke?

it means their servers were unreachable due to network misconfig.

Re: Elevated errors across many models

#112

Post by bc569a80a344f9c »

wouldbecouldbe wrote: post #109
l1n wrote: post #96

Earlier quoted context omitted.

Also an engineer on this incident. This was a network routing misconfiguration - an overlapping route advertisement caused traffic to some of our inference backends to be blackholed. Detection took longer than we’d like (about 75 minutes from impact to identification), and some of our normal mitigation paths didn’t work as expected during the incident. The bad route has been removed and service is restored. We’re doi…

Trying to understand what this means. Did the bad route cause an overload? Was there a code error on that route that wasn’t spotted? Was it a code issue or an instance that broke?

It says network routing issue.

Network routes consist of a network (a range of IPs) and a next hop to send traffic for that range to.

These can overlap. Sometimes that’s desirable, sometimes it is not. When routers have two routes that are exactly the same they often load balance (in some fairly dumb, stateless fashion) between possible next hops, when one of the routes is more specific, it wins.

Routes get injected by routers saying “I am responsible for this range” and setting themselves as the next hop, others routers that connect to them receive this advertisement and propagate it to their own router peers further downstream.

An example would be advertising 192.168.0.0/23, which is the range of 192.168.0.0-192.168.1.255.

Let’s say that’s your inference backend in some rows in a data center.

Then, through some misconfiguration, some other router starts announcing 192.168.1.0/24 (192.168.1.0-192.168.1.255). This is more specific, that traffic gets sent there, and half of the original inference pod is now unreachable.

Re: Elevated errors across many models

#114

Post by Buttons840 »

dinkleberg wrote: post #18

Props to them for actually updating their status page as issues are happening rather than hours later. I was working with claude code and hit an API error, checked the status page and sure enough there was an outage. This should be a given for any service that others rely on, but sadly this is seldom the case.

"There's a problem and we already know about it" is so much better than "there's a problem and we don't know about it and/or are hoping it will magically go away and that we won't be embarrassed".

Re: Elevated errors across many models

#115

Post by manuisin »

manquer wrote: post #53
sebastiennight wrote: post #51

In the Claude.ai chat, this was announced to me as "You have reached the messages quota for your account. It will reset in 2 hours, or you can upgrade now" Either I have perfect timing for reaching my quota limits, or some product monetization manager deserves a raise.

More likely that error handling is not well implemented - i.e Either backend is not throwing the equivalent of 429/402 errors or the gateway is not handling the errors well and returns this message even though a 429 is being thrown.

This sort of thing keeps me skeptical of AI quite a bit. ChatGPT also has non sensical errors messages for random failures, Gemini too. These companies have infinite compute and yet they haven't been able to implement reliable/graceful error handling in 2+ years for a chat app? Why are they promising us they can replace all developers?

Re: Elevated errors across many models

#116

Post by disqard »

bc569a80a344f9c wrote: post #112
wouldbecouldbe wrote: post #109

Earlier quoted context omitted.

Trying to understand what this means. Did the bad route cause an overload? Was there a code error on that route that wasn’t spotted? Was it a code issue or an instance that broke?

It says network routing issue. Network routes consist of a network (a range of IPs) and a next hop to send traffic for that range to. These can overlap. Sometimes that’s desirable, sometimes it is not. When routers have two routes that are exactly the same they often load balance (in some fairly dumb, stateless fashion) between possible next hops, when one of the routes is more specific, it wins. Routes get injected…

Thank you for that explanation!

Re: Elevated errors across many models

#118

Post by cevn »

LanceH wrote: post #49
dinkleberg wrote: post #18

Props to them for actually updating their status page as issues are happening rather than hours later. I was working with claude code and hit an API error, checked the status page and sure enough there was an outage. This should be a given for any service that others rely on, but sadly this is seldom the case.

Confusingly, I was trying to debug something with a 529, and this outage really had me going for a minute.

The 529 is coming from inside the house?!

Re: Elevated errors across many models

#119

Post by tayo42 »

l1n wrote: post #96
palcu wrote: post #58

Hello, I'm one of the engineers who worked on the incident. We have mitigated the incident as of 14:43 PT / 22:43 UTC. Sorry for the trouble.

Also an engineer on this incident. This was a network routing misconfiguration - an overlapping route advertisement caused traffic to some of our inference backends to be blackholed. Detection took longer than we’d like (about 75 minutes from impact to identification), and some of our normal mitigation paths didn’t work as expected during the incident. The bad route has been removed and service is restored. We’re doi…

I was kind surprised to see details like that in a comment, but clicked on your personal website and see your a Co-founder, so I guess no one is going to repremand you lol

Re: Elevated errors across many models

#120

Post by giancarlostoro »

palcu wrote: post #58

Hello, I'm one of the engineers who worked on the incident. We have mitigated the incident as of 14:43 PT / 22:43 UTC. Sorry for the trouble.

Any chance you guys could do write ups on these incidents similar to how CloudFlare does? For all the heat some people give them, I trust CloudFlare more with my websites than a lot of other companies because of their dedication to transparency.
Post reply on HN