Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Level 3 Global Outage

puck.nether.net

Post reply on HN
Print view View original

181–190 of 393 posts

Re: Level 3 Global Outage

#181

Post by itguy4321365 »

ramshanker wrote: post #94

I hope these kind of “ipv4” only outages encourages more and more websites to upgrade to ipv6. #OutageBenefit ;)

This doesn't have anything to do with IPv4 vs IPv6. It is a routing issue with BGP. To give an analogy,

if every website were a house, and every house has a house number (IP address-- either IPv4 or IPv6), and a group of houses form cities and towns that can be identified by a number (AS/ Atonomous System number), the highways between cities are similar to BGP routes, and if half of the world's internet traffic goes through the city of Centurylink (AS3356),

If the city of CenturyLink (AS3356) shut down traffic, either on purpose or on accident.

...then it doesn't matter if your house number / IP address is a 32bit number or a 128bit number because traffic needs to take a different route.

This is what everyone is worried about BGP routes, not IP addresses.

Re: Level 3 Global Outage

#183

Post by swinglock »

josephb wrote: post #84
iso1210 wrote: post #79

Looks like Centurylink/Level3 (as3356) might not be withdrawing routes after people close their peering?

That's what various networks have reported. It kind of makes it hard to route around an upstream, if they keep announcing your routes even when there isn't a path to you!

Quick hack; split all your announcements in two, making the new announcement route around their old stale announcement by being more specific.

Re: Level 3 Global Outage

#184

Post by matsemann »

djxfade wrote: post #154
tyfon wrote: post #143

Seems like "the internet" works again here in Norway. I've been limited to local sites all day. Hacker news has been off for several hours for me. Whatever it was it must have been nasty.

I had the same issue on my fiber connection (Altibox/BKK), however, no problems on my mobile using 4G (Dipper/Telenor)

I couldn't reach HN on neither Altibox or 4g/telenor.

Re: Level 3 Global Outage

#185

Post by emilstahl »

kitteh wrote: post #179
kitteh wrote: post #155

Earlier quoted context omitted.

Most of level3s settlement free peers aka "tier 1s" have shutdown or depreffed their sessions with them. Example: https://mobile.twitter.com/TeliaCarrier/status/1300074378378...

Root cause identified. Folks are turning things back on now.

Source?

Re: Level 3 Global Outage

#188

Post by beagle3 »

kitteh wrote: post #124

Massive reconvergence event in their network, causing edge router bgp sessions to bounce (due to cpu). Right now all their big peers are shutting down sessions with them to give level3s network the ability to reconverge. Prefixes announced to 3356 are frozen on their route reflectors and not getting withdrawn. Edit: if you are a Level3 customer shut your sessions down to them.

History doesn't repeat, but it rhymes ....

There was a huge AT&T outage in 1990 that cut off most US long distance telephony (which was, at the time, mostly "everything not within the same area code").

It was a bug. It wasn't a reconvergence event, but it was a distant cousin: Something would cause a crash; exchanges would offload that something to other exchanges, causing them to crash -- but with enough time for the original exchange to come back up, receive the crashy event back, and crash again.

The whole network was full of nodes crashing, causing their peers to crash, ad infinitum. In order to bring the network back up, they needed to either take everything down at the same time (and make sure all the queues are emptied), but even that wouldn't have made it stable, because a similar "patient 0" event would have brought the whole network down.

Once the problem was understood, they reverted to an earlier version which didn't have the bug, and the network re-stabilized.

The lore I grew up on is that this specific event was very significant in pushing and funding research into robust distributed systems, of which the best known result is Erlang and its ecosystem - originally built, and still mostly used, to make sure that phone exchanges don't break.

[0] https://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collap...

Re: Level 3 Global Outage

#190

Post by [deleted] »

jsjohnst wrote: post #176
[deleted] wrote: post #171

Earlier quoted context omitted.

[deleted]

> My current theory: The current Internet with the crappy BGP security model is the way it is because it was built by network admins turned amateur software developers. We need raise the level of software quality in the core parts of the net. Further proving my point. Thanks and have a great day!

[deleted]
Post reply on HN