Live data from Hacker News

Level 3 Global Outage

puck.nether.net

221–230 of 393 posts

Re: Level 3 Global Outage

#221

Has anyone any good resources for learning more about the "internet-level" infrastructure affected today and how global networks are connected?

https://mobile.twitter.com/Level3 (not an internet level, just a company :)

Were their tweets protected (i.e. only visible to approved followers) when you posted that link, or is that in response to this event?

Re: Level 3 Global Outage

#222
post #124

Massive reconvergence event in their network, causing edge router bgp sessions to bounce (due to cpu). Right now all their big peers are shutting down sessions with them to give level3s network the ability to reconverge. Prefixes announced to 3356 are frozen on their route reflectors and not getting withdrawn. Edit: if you are a Level3 customer shut your sessions down to them.

What is a reconvergence event? Is that what's described in your last sentence?

BGP is a path-vector routing protocol, every router on the internet is constantly updating its routing tables based on paths provided by its peers to get the shortest distance to an advertised prefix. When a new route is announced it takes time to propagate through the network and for all routers in the chain to “converge” into a single coherent view.

If this is indeed a reconvergence event, that would imply there’s been a cascade of route table updates that have been making their way through CTL/L3’s network - meaning many routers are missing the “correct” paths to prefixes and traffic is not going where it is supposed to, either getting stuck in a routing loop or just going to /dev/null because the next hop isn’t available.

This wouldn’t be such a huge issue if downstream systems could shut down their BGP sessions with CTL and have traffic come in via other routes, but doing so is not resulting in the announcements being pulled from the Level 3 AS - something usually reflective of the CPU on the routers being overloaded processing route table updates or an issue with the BGP communication between them.

Convergence time is a known bugbear of BGP.

Re: Level 3 Global Outage

#223

Has anyone any good resources for learning more about the "internet-level" infrastructure affected today and how global networks are connected?

"An Open Platform to Teach How the Internet Practically Works" from NANOG perhaps:

* https://www.youtube.com/watch?v=8SRjTqH5Z8M

The Network Startup Resource Center out of UOregen has some good tutorials on BGP and connecting networks owned by different folks:

* https://learn.nsrc.org/bgp

NANOG also has a lot of good videos on their channel from their conferences, including one on optical fibre if you want to get into the low-level ISO Layer 1 stuff:

* https://www.youtube.com/watch?v=nKeZaNwPKPo

In a similar vein, NANOG "Panel: Demystifying Submarine Cables"

* https://www.youtube.com/watch?v=Pk1e2YLf5Uc

Re: Level 3 Global Outage

#224
post #208

Has anyone any good resources for learning more about the "internet-level" infrastructure affected today and how global networks are connected?

Unfortunately, this infrastructure is at an uncanny intersection of technology, business and politics. To learn the technical aspect of it, you can follow any network engineering certification materials or resources that delve into dynamic routing protocols, notably BGP. Inter-ISP networking is nothing but setting up BGP sessions and filters at the technical level. Why you set these up, and under what conditions is a…

Judging by other comments, it seems there's a space to fill this niche with a series of blog articles or a book, if you're that sort of person.

Re: Level 3 Global Outage

#225
How the xxxx did it take CenturyLink/Level3 like 3-4 hours to fix this problem?

Again (https://news.ycombinator.com/item?id=24322988) not a network engineer, but it seemed like their routers actively stopped other networks from working around the problem since L3 would still keep pushing other networks' old routes, even after those networks tried to stop that.

Also: BGP probably needs to redesigned from the ground up by software engineers with experience from designing systems that can remain working with hostile actors.

Re: Level 3 Global Outage

#226

CenturyLink/Level3 on Twitter: "We are able to confirm that all services impacted by today’s IP outage have been restored. We understand how important these services are to our customers, and we sincerely apologize for the impact this outage caused." https://twitter.com/CenturyLink/status/1300089110858797063

I hope they provide a root cause analysis

Re: Level 3 Global Outage

#227
post #56
post #55

Internet infrastructure is broken. Why do a few companies control the backbone of the internet? Shouldn’t there be a fallback or disaster recovery plan if one or more of these companies become unavailable?

Why doesn't stuff just route around this automatically, if one provider has problems?

The problem is the provider having problems is still sending misconfigured routes after the other providers have tried to pull them in response to the outage. So it’s as if CenturyLink was doing a massive BGP attack against their peers, pointing at a black hole.

Re: Level 3 Global Outage

#228
post #208

Has anyone any good resources for learning more about the "internet-level" infrastructure affected today and how global networks are connected?

Unfortunately, this infrastructure is at an uncanny intersection of technology, business and politics. To learn the technical aspect of it, you can follow any network engineering certification materials or resources that delve into dynamic routing protocols, notably BGP. Inter-ISP networking is nothing but setting up BGP sessions and filters at the technical level. Why you set these up, and under what conditions is a…

> or even just getting an ASN and some IPv6 PA space and trying to announce it somewhere

That’s fairly expensive to do just for a hobby interest, but at least the price has came down since I last looked.

Re: Level 3 Global Outage

#229
post #158
post #156

Earlier quoted context omitted.

You were repeating a suspicion as if it was yours, as if it was a shared view in that thread, it wasn't, i'm not a network engineer but I read the thread too, nobody needs people spreading misinformation in a crisis just to sound smart, its not useful and usually harmful. No superiority here.

> You were repeating a suspicion as if it was yours That is a misrepresentation. I wrote: "Not a network engineer, but based on the comments there". I prefaced the insecure speculation part with "possibly". It was obvious to any reader this was a summary of the emails there. You are overreacting.

> You're not a network engineer but it looks like a blackhole incident to you?

I'm the one overreacting?

All I did was ask you if it was your opinion or not.

Re: Level 3 Global Outage

#230
post #208

Earlier quoted context omitted.

Unfortunately, this infrastructure is at an uncanny intersection of technology, business and politics. To learn the technical aspect of it, you can follow any network engineering certification materials or resources that delve into dynamic routing protocols, notably BGP. Inter-ISP networking is nothing but setting up BGP sessions and filters at the technical level. Why you set these up, and under what conditions is a…

> or even just getting an ASN and some IPv6 PA space and trying to announce it somewhere That’s fairly expensive to do just for a hobby interest, but at least the price has came down since I last looked.

A RIPE ASN (as a end-user through a LIR) and PA v6 will cost you around $100 per year and some mild paperwork, there's plenty of companies/organizations that will help you with that (shameless plug: bgp.wtf, that's us).

Afterwards, announcing this space is probably the cheapest with vultr (but their BGP connectivity to VMs is uh, erratic at times) or with ix-vm.cloud, or with packet.net (more expensive). You can also try to colo something in he.net FMT2 and reach FCIX, or something in kleyrex in Germany. All in all, you should be able to do something like run a toy v6 anycast CDN at not more than $100 per month.

Post reply on HN