Source https://puck.nether.net/pipermail/outages/2020-August/013229...
Level 3 Global Outage
321–330 of 393 posts
Re: Level 3 Global Outage
#322Based on what I've seen: They essentially "shut down the Internet" for probably a quarter of the global population for about 3-4 hours. That response time is atrocious. It wasn't that they needed to fix broken hardware, rather they needed to stop running hardware from actively sabotaging the global routing via the inherently insecure BGP protocol. That took 3-4 hours to happen. As an example: Being in Sweden with an…
Seems they didn't even get around to doing so, rather asking other carriers to stop peering with them. https://twitter.com/TeliaCarrier/status/1300074378378518528?...
Re: Level 3 Global Outage
#323Earlier quoted context omitted.
Which channels
They wouldn't be non-public if they told us plebs
Re: Level 3 Global Outage
#324Earlier quoted context omitted.
History doesn't repeat, but it rhymes .... There was a huge AT&T outage in 1990 that cut off most US long distance telephony (which was, at the time, mostly "everything not within the same area code"). It was a bug. It wasn't a reconvergence event, but it was a distant cousin: Something would cause a crash; exchanges would offload that something to other exchanges, causing them to crash -- but with enough time for th…
The thing with feature group D trunks to the long distance network is you could (and still can on non-IP/mobile networks) manually route to another long distance carrier like Verizon, and sidestep the outage from the subscriber end, full stop. That's certainly not possible with any of the contemporary internet outages.
Re: Level 3 Global Outage
#325Summary: On August 30, 2020 10:04 GMT, CenturyLink identified an issue to be affecting users across multiple markets. The IP Network Operations Center (NOC) was engaged, and initial research identified that an offending flowspec announcement prevented Border Gateway Protocol (BGP) from establishing across multiple elements throughout the CenturyLink Network. The IP NOC deployed a global configuration change to block…
Its a super useful tool if you want to blast out an ACL across your network in seconds (using BGP) but it has a number of sharp edges. Several networks, including Cloudflare have learned what it can do. I've seen a few networks basically blackhole traffic or even lock themselves out of routers due to a poorly made Flowspec rules or a bug in the implementation.
Re: Level 3 Global Outage
#326Earlier quoted context omitted.
The thing with feature group D trunks to the long distance network is you could (and still can on non-IP/mobile networks) manually route to another long distance carrier like Verizon, and sidestep the outage from the subscriber end, full stop. That's certainly not possible with any of the contemporary internet outages.
you can inject changes in routing, but if the other other carrier doesn't route around the affected network, you're back to square one. That's part of why Level3/CenturyLink was depeered and why several prefixes that are normally announced through it were quickly rerouted by owners.
Re: Level 3 Global Outage
#327Earlier quoted context omitted.
Having a single, cryptographically assured source of truth for routing data is a turnkey censorship nightmare waiting to happen. All it takes is a national military to care enough to put pressure on the database operator, legal or otherwise, and suddenly your legitimate routes are no longer accepted. If you think this wouldn't be used to shut down things like future Snowden-style leaks or Wikileaks or The Shadow Brok…
sneak you should come back to irc :)
Re: Level 3 Global Outage
#328Learn from Feynman. Explain things using concepts the target audience can be expected to understand.
Real mastery of a concept is when you can explain it using simple terms to any other reasonably intelligent person.
Re: Level 3 Global Outage
#329Observation: You're not really adjusting your language to the target audience. (Edited away potentially hurtful language.) Learn from Feynman. Explain things using concepts the target audience can be expected to understand. Real mastery of a concept is when you can explain it using simple terms to any other reasonably intelligent person.
A few years ago folks wanted to bake in additional functionality. For example, packet filters (aka ACLs) normally are deployed to router configuration files using each operators own tooling. To deploy this against hundreds or thousands of routers rapidly was a challenge for them (not good at swdev, etc.). So the idea was we already have a protocol that propagates state to every router rapidly in the network, let's find a way to bake ACLs into the BGP updates.
The result wasnt that good for a few reasons: 1) bgp state isn't sticky. If a router goes offline or bgp sessions reset, acls go away. That means if you are using flowspec for a critical need like always on packet filters you've got the wrong tool. 2) the implementation had various bugs. 3) most importantly it gave people a really easy way to hurt themselves globally. There was no phased deployment with pre and post checks. What you deployed led to packet filters being installed across the network in seconds. In most cases (depends on your config) the only way to remove it is remove the specific flowspec route or have bgp reset to it.
I've seen bad flowspec routes core dump the daemon on a router responsible for programming ACLs that led to them being unable to withdraw the programmed entry. I've seen as bugs on tcp/UDP port matches go wrong and eat lot more than intended. I've seen so many flowspec rules installed on a network where it exhausted routers ability to inspect and process packets and you'd see flat lining of packets being dropped.
In my opinion, it's a hack around not having a good ACL deployment tool that has led to many outages in its wake.
Edit: another flowspec gotcha. Some folks like to integrate ddos tooling systems into flowspec. An example of this is if I run a network and some IP address behind me gets lit up, deploy a rule for that specific IP and rate limit traffic to it. Unfortunately, sometimes folks don't put a lot of care into making sure it can't mess with internal IPs that should be off limits. Like route reflectors, router loopback IPs, etc. I've seen situations where some networks have had a bad day due to a ddos or traffic mis classified as ddos by auto installing rules to protect something but actually impair legitimate communications to network infrastructure which then causes the outage.
Also, flowspec doesn't work like regular ACLs where you have input and output on a per interface basis - it applies to all traffic traversing a router, which makes it difficult to say which interfaces should be exempt (think internal vs external).