Has anyone any good resources for learning more about the "internet-level" infrastructure affected today and how global networks are connected?
https://mobile.twitter.com/Level3 (not an internet level, just a company :)
Level 3 Global Outage
221–230 of 393 posts
Re: Level 3 Global Outage
#222Massive reconvergence event in their network, causing edge router bgp sessions to bounce (due to cpu). Right now all their big peers are shutting down sessions with them to give level3s network the ability to reconverge. Prefixes announced to 3356 are frozen on their route reflectors and not getting withdrawn. Edit: if you are a Level3 customer shut your sessions down to them.
What is a reconvergence event? Is that what's described in your last sentence?
If this is indeed a reconvergence event, that would imply there’s been a cascade of route table updates that have been making their way through CTL/L3’s network - meaning many routers are missing the “correct” paths to prefixes and traffic is not going where it is supposed to, either getting stuck in a routing loop or just going to /dev/null because the next hop isn’t available.
This wouldn’t be such a huge issue if downstream systems could shut down their BGP sessions with CTL and have traffic come in via other routes, but doing so is not resulting in the announcements being pulled from the Level 3 AS - something usually reflective of the CPU on the routers being overloaded processing route table updates or an issue with the BGP communication between them.
Convergence time is a known bugbear of BGP.
Re: Level 3 Global Outage
#223Has anyone any good resources for learning more about the "internet-level" infrastructure affected today and how global networks are connected?
* https://www.youtube.com/watch?v=8SRjTqH5Z8M
The Network Startup Resource Center out of UOregen has some good tutorials on BGP and connecting networks owned by different folks:
NANOG also has a lot of good videos on their channel from their conferences, including one on optical fibre if you want to get into the low-level ISO Layer 1 stuff:
* https://www.youtube.com/watch?v=nKeZaNwPKPo
In a similar vein, NANOG "Panel: Demystifying Submarine Cables"
Re: Level 3 Global Outage
#224Has anyone any good resources for learning more about the "internet-level" infrastructure affected today and how global networks are connected?
Unfortunately, this infrastructure is at an uncanny intersection of technology, business and politics. To learn the technical aspect of it, you can follow any network engineering certification materials or resources that delve into dynamic routing protocols, notably BGP. Inter-ISP networking is nothing but setting up BGP sessions and filters at the technical level. Why you set these up, and under what conditions is a…
Re: Level 3 Global Outage
#225Again (https://news.ycombinator.com/item?id=24322988) not a network engineer, but it seemed like their routers actively stopped other networks from working around the problem since L3 would still keep pushing other networks' old routes, even after those networks tried to stop that.
Also: BGP probably needs to redesigned from the ground up by software engineers with experience from designing systems that can remain working with hostile actors.
Re: Level 3 Global Outage
#226CenturyLink/Level3 on Twitter: "We are able to confirm that all services impacted by today’s IP outage have been restored. We understand how important these services are to our customers, and we sincerely apologize for the impact this outage caused." https://twitter.com/CenturyLink/status/1300089110858797063
Re: Level 3 Global Outage
#227Internet infrastructure is broken. Why do a few companies control the backbone of the internet? Shouldn’t there be a fallback or disaster recovery plan if one or more of these companies become unavailable?
Why doesn't stuff just route around this automatically, if one provider has problems?
Re: Level 3 Global Outage
#228Has anyone any good resources for learning more about the "internet-level" infrastructure affected today and how global networks are connected?
Unfortunately, this infrastructure is at an uncanny intersection of technology, business and politics. To learn the technical aspect of it, you can follow any network engineering certification materials or resources that delve into dynamic routing protocols, notably BGP. Inter-ISP networking is nothing but setting up BGP sessions and filters at the technical level. Why you set these up, and under what conditions is a…
That’s fairly expensive to do just for a hobby interest, but at least the price has came down since I last looked.
Re: Level 3 Global Outage
#229Earlier quoted context omitted.
You were repeating a suspicion as if it was yours, as if it was a shared view in that thread, it wasn't, i'm not a network engineer but I read the thread too, nobody needs people spreading misinformation in a crisis just to sound smart, its not useful and usually harmful. No superiority here.
> You were repeating a suspicion as if it was yours That is a misrepresentation. I wrote: "Not a network engineer, but based on the comments there". I prefaced the insecure speculation part with "possibly". It was obvious to any reader this was a summary of the emails there. You are overreacting.
I'm the one overreacting?
All I did was ask you if it was your opinion or not.
Re: Level 3 Global Outage
#230Earlier quoted context omitted.
Unfortunately, this infrastructure is at an uncanny intersection of technology, business and politics. To learn the technical aspect of it, you can follow any network engineering certification materials or resources that delve into dynamic routing protocols, notably BGP. Inter-ISP networking is nothing but setting up BGP sessions and filters at the technical level. Why you set these up, and under what conditions is a…
> or even just getting an ASN and some IPv6 PA space and trying to announce it somewhere That’s fairly expensive to do just for a hobby interest, but at least the price has came down since I last looked.
Afterwards, announcing this space is probably the cheapest with vultr (but their BGP connectivity to VMs is uh, erratic at times) or with ix-vm.cloud, or with packet.net (more expensive). You can also try to colo something in he.net FMT2 and reach FCIX, or something in kleyrex in Germany. All in all, you should be able to do something like run a toy v6 anycast CDN at not more than $100 per month.