WAN router IP address change blamed for global Microsoft 365 outage
1–10 of 68 posts
Re: WAN router IP address change blamed for global Microsoft 365 outage
#2Having, uh, had bad things happen with router configuration I feel for them.
https://blog.cloudflare.com/cloudflare-outage-on-july-17-202...
Re: WAN router IP address change blamed for global Microsoft 365 outage
#3Having, uh, had bad things happen with router configuration I feel for them. https://blog.cloudflare.com/cloudflare-outage-on-july-17-202...
It does seem like network configuration remains rather manual compared to other large scale systems that include more automation.
In Microsoft's case, the remediation is not to put in place higher level systems to safely accomplish the goal of the command. Instead:
- "We have blocked highly impactful commands from getting executed on the devices (Completed)"
- "We will require all command execution on the devices to follow safe change guidelines (Estimated completion: February 2023)"
Requiring commands to follow guidelines sounds suspiciously like they're requiring network ops not to break things.
Re: WAN router IP address change blamed for global Microsoft 365 outage
#4Having, uh, had bad things happen with router configuration I feel for them. https://blog.cloudflare.com/cloudflare-outage-on-july-17-202...
It does seem like network configuration remains rather manual compared to other large scale systems that include more automation. In Microsoft's case, the remediation is not to put in place higher level systems to safely accomplish the goal of the command. Instead: - "We have blocked highly impactful commands from getting executed on the devices (Completed)" - "We will require all command execution on the devices to…
That's the norm in network ops. Automated testing is pretty much impossible, easy rollback may be possible depending on exactly what was screwed, but not always.
Take this for example, looks like the problem was an unplanned recalculation of routing tables. That's not going to be the case on a small scale test network, and rolling back won't help, indeed in this case it likely would cause more problems.
Re: WAN router IP address change blamed for global Microsoft 365 outage
#5Having, uh, had bad things happen with router configuration I feel for them. https://blog.cloudflare.com/cloudflare-outage-on-july-17-202...
Holy shit I have been there and it sucks. I wasn't the guy who made the change, but I was on the long call that followed.
Re: WAN router IP address change blamed for global Microsoft 365 outage
#6> As part of a planned change to update the IP address on a WAN router, a command given to the router caused it to send messages to all other routers in the WAN, which resulted in all of them recomputing their adjacency and forwarding tables. During this re-computation process, the routers were unable to correctly forward packets traversing them. "The command that caused the issue has different behaviors on different network devices, and the command had not been vetted using our full qualification process on the router on which it was executed."
From this it sounds like they might have changed the primary loopback IP, which by default is the "router-id" for various routing protocols, causing the entire network to have to reconverge. You can override the default router-id with an explicit address that does not depend on lo0 but lots of networks don't do that.
It's extremely uncommon to change the primary loopback address. It's less uncommon to add an additional one but as the article says that syntax varies by vendor: Juniper will add as additional by default, Cisco and Arista will replace the existing primary one (IPv4) unless you include the "secondary" keyword...
Re: WAN router IP address change blamed for global Microsoft 365 outage
#7Earlier quoted context omitted.
It does seem like network configuration remains rather manual compared to other large scale systems that include more automation. In Microsoft's case, the remediation is not to put in place higher level systems to safely accomplish the goal of the command. Instead: - "We have blocked highly impactful commands from getting executed on the devices (Completed)" - "We will require all command execution on the devices to…
That's the norm in network ops. Automated testing is pretty much impossible, easy rollback may be possible depending on exactly what was screwed, but not always. Take this for example, looks like the problem was an unplanned recalculation of routing tables. That's not going to be the case on a small scale test network, and rolling back won't help, indeed in this case it likely would cause more problems.
One of the reasons I got out of network engineering was how frequently the work I was required to do would cause unintended consequences. You can do all your due diligence, get your work blessed by vendor support, and still get blown up by a bug or undocumented behaviors on a regular basis. The conspiratorial part of my brain says these network device makers intentionally provide unreliable software and terrible documentation to bolster their support contract profits. I was just the guy typing in the commands and getting all the blame.
Re: WAN router IP address change blamed for global Microsoft 365 outage
#8Having, uh, had bad things happen with router configuration I feel for them. https://blog.cloudflare.com/cloudflare-outage-on-july-17-202...
Holy shit I have been there and it sucks. I wasn't the guy who made the change, but I was on the long call that followed.
Time to share one of my favorite talks (and speakers) ever -
"Debugging Under Fire: Keep your Head when Systems have Lost their Mind" (Bryan Cantrill, GOTO 2017)
Re: WAN router IP address change blamed for global Microsoft 365 outage
#9It seems that in modern large scale systems networking continues to be one of the few things were a a seemingly small and inconsequential change can cause entire cloud providers and highly redundant systems to go down. It makes sense as networking is the fabric connecting all systems together but each time an incident like this occurs I'm reminded of just how important networking is.
Network engineers and the people handling network ops always amaze me.
Re: WAN router IP address change blamed for global Microsoft 365 outage
#10Having, uh, had bad things happen with router configuration I feel for them. https://blog.cloudflare.com/cloudflare-outage-on-july-17-202...
How about describing how you implement systems that prevent this? You kind of talk about what was 'fixed', but not how. CI/CD is pretty hard to do for global networking changes. I'm sure whatever CF has done in this area is a lot of magic sauce and it would be super interesting to learn more about it, even at a high level.