Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Level 3 Global Outage

puck.nether.net

Post reply on HN
Print view View original

301–310 of 393 posts

Re: Level 3 Global Outage

#301

Post by dadver »

beagle3 wrote: post #188
kitteh wrote: post #124

Massive reconvergence event in their network, causing edge router bgp sessions to bounce (due to cpu). Right now all their big peers are shutting down sessions with them to give level3s network the ability to reconverge. Prefixes announced to 3356 are frozen on their route reflectors and not getting withdrawn. Edit: if you are a Level3 customer shut your sessions down to them.

History doesn't repeat, but it rhymes .... There was a huge AT&T outage in 1990 that cut off most US long distance telephony (which was, at the time, mostly "everything not within the same area code"). It was a bug. It wasn't a reconvergence event, but it was a distant cousin: Something would cause a crash; exchanges would offload that something to other exchanges, causing them to crash -- but with enough time for th…

This sounds like the event that is described in the book Masters of Deception: The gang that ruled cyberspace. The way I remember it the book attributes the incident to MoD, while of course still being the result of a bug/faulty design.

Re: Level 3 Global Outage

#302

Post by rootsudo »

colde wrote: post #261
ystad wrote: post #226

Earlier quoted context omitted.

I hope they provide a root cause analysis

Based on experience it will probably not public, or at least very limited. But customers are likely to get one, at least if they request it.

Being it was pretty big, they'll probably make it public.

Re: Level 3 Global Outage

#303

Post by rolph »

dudus wrote: post #286
xwdv wrote: post #271

Earlier quoted context omitted.

What are the right IRC channels?

I believe these are mostly non public channels where backbone and network infrastructure engineers from different companies congregate to discuss outages like this.

also channels where hats of various type discuss advantages opportunities and challenges presented by such outages

Re: Level 3 Global Outage

#304

Post by jlgaddis »

ystad wrote: post #226
emilstahl wrote: post #215

CenturyLink/Level3 on Twitter: "We are able to confirm that all services impacted by today’s IP outage have been restored. We understand how important these services are to our customers, and we sincerely apologize for the impact this outage caused." https://twitter.com/CenturyLink/status/1300089110858797063

I hope they provide a root cause analysis

https://news.ycombinator.com/item?id=24324280

Re: Level 3 Global Outage

#305

Post by rglover »

This knocked out the Starbucks app and some of their systems this morning. A bunch of people in line couldn't log in and they were saying parts of their whole internal system were down, too.

Re: Level 3 Global Outage

#306

Post by stoops »

rolph wrote: post #279
hpkuarg wrote: post #274

Earlier quoted context omitted.

Second time in this thread you've mentioned the right IRC channels. Does one need to invoke some secret code to find out what they are? :-)

in certain instances, yes depending on the nature or subject matter of the channel ok lets go for broke, there are a LOT of clandestine IRC servers and exclusive gatekeeping of channels. you wont know about unless you have an IRL reference

The first rule of Fight Club is that you don't talk about Fight Club.

Re: Level 3 Global Outage

#307

Post by FR10 »

hinkley wrote: post #299
cyphar wrote: post #159

Earlier quoted context omitted.

All professional chess games have a time limit for each player (if you've ever heard of "chess clocks" -- that's what they're used for). In "slow chess" each player has a 2-hour limit and all of the other time control schemes (such as rapid and blitz) are much shorter.

There’s an interesting protocol for splitting a Go or chess game over multiple days so that neither party has the entire time to think about their response to the last move: at the end of the day the final move is made by one player but is sealed, not to be revealed until the start of the next session. For this to work on an internet competition, the judges would need a backup, possibly very low bandwidth communicati…

Yes, this is call Adjournment[0] and they used to do it until 20 or so years ago when computer analysis became too good/mainstream.

[0] https://en.wikipedia.org/wiki/Adjournment_(games)

Re: Level 3 Global Outage

#309

Post by Yajirobe »

kitteh wrote: post #124

Massive reconvergence event in their network, causing edge router bgp sessions to bounce (due to cpu). Right now all their big peers are shutting down sessions with them to give level3s network the ability to reconverge. Prefixes announced to 3356 are frozen on their route reflectors and not getting withdrawn. Edit: if you are a Level3 customer shut your sessions down to them.

I know some of these words

Re: Level 3 Global Outage

#310

Post by doctorshady »

beagle3 wrote: post #188
kitteh wrote: post #124

Massive reconvergence event in their network, causing edge router bgp sessions to bounce (due to cpu). Right now all their big peers are shutting down sessions with them to give level3s network the ability to reconverge. Prefixes announced to 3356 are frozen on their route reflectors and not getting withdrawn. Edit: if you are a Level3 customer shut your sessions down to them.

History doesn't repeat, but it rhymes .... There was a huge AT&T outage in 1990 that cut off most US long distance telephony (which was, at the time, mostly "everything not within the same area code"). It was a bug. It wasn't a reconvergence event, but it was a distant cousin: Something would cause a crash; exchanges would offload that something to other exchanges, causing them to crash -- but with enough time for th…

The thing with feature group D trunks to the long distance network is you could (and still can on non-IP/mobile networks) manually route to another long distance carrier like Verizon, and sidestep the outage from the subscriber end, full stop. That's certainly not possible with any of the contemporary internet outages.
Post reply on HN