Having, uh, had bad things happen with router configuration I feel for them. https://blog.cloudflare.com/cloudflare-outage-on-july-17-202...
Your CEO sure doesn't seem to have much empathy when it's someone else though:
WAN router IP address change blamed for global Microsoft 365 outage
31–40 of 68 posts
Re: WAN router IP address change blamed for global Microsoft 365 outage
#32Earlier quoted context omitted.
This may be true of enterprise network engineers but I’ve worked across a lot of very large networks (telco, not cloud) and we never ever trust the vendor. The kind of bugs that I’ve read about in errata notes over the years is wild and truly unpredictable.
It wouldn't be the first time that your redundant vendors end up sharing a conduit for a bunch of fiber somewhere. Guess where that backhoe will start digging?
Redundant vendors in the GP’s context referred to using multiple router vendors, eg Cisco and Juniper.
Using multiple connectivity vendors doesn’t guarantee path diversity. Demanding fibre maps and ensuring that your connectivity has separate points of entry into the building, doesn’t cross outside the building, and validating with your DC provider that your cross connects aren’t crossing either, guaranteed path diversity / redundancy.
Re: WAN router IP address change blamed for global Microsoft 365 outage
#33Earlier quoted context omitted.
One of the reasons I got out of network engineering was how frequently the work I was required to do would cause unintended consequences. You can do all your due diligence, get your work blessed by vendor support, and still get blown up by a bug or undocumented behaviors on a regular basis. The conspiratorial part of my brain says these network device makers intentionally provide unreliable software and terrible docu…
I remember the first time I got access to an employers production Cisco router. It’s pretty scary how easy it is to majorly fuck something up. There isn’t a concept of a transaction or a rollback. You just enter a command, press enter and it’s live. To counter this we’d write all the commands we planned on executing and peer review it. Nothing was to be done “on the fly” (at least in theory) In short, coming from a d…
>"There isn’t a concept of a transaction or a rollback. You just enter a command, press enter and it’s live."
This hasn't been true for a very long time. Juniper router's have rollbacks, commits and revisions:
https://www.juniper.net/documentation/us/en/software/junos/c...
and
https://www.juniper.net/documentation/us/en/software/junos/c...
Cisco has similar:
https://www.cisco.com/c/en/us/td/docs/ios/ios_xe/fundamental...
Re: WAN router IP address change blamed for global Microsoft 365 outage
#34Earlier quoted context omitted.
That's the norm in network ops. Automated testing is pretty much impossible, easy rollback may be possible depending on exactly what was screwed, but not always. Take this for example, looks like the problem was an unplanned recalculation of routing tables. That's not going to be the case on a small scale test network, and rolling back won't help, indeed in this case it likely would cause more problems.
> That's the norm in network ops. Automated testing is pretty much impossible, easy rollback may be possible depending on exactly what was screwed, but not always. Ansible/Napalm is a thing in NetOps in some places. Some folks use Eve-ng / GNS3 to spin up virtual networks to test config changes, and it may be possible to do CI/CD changes if you track things in a repo. Juniper JunOS has auto-rollback if you don't conf…
commit confirmed is such a life-saver. I ran a production network which spanned multiple continents and even though I probably only ever actually needed commit confirmed a single digit number of times, the fact that it was there made every change I did 99% less stressful. I knew that even if I made a mistake, all I had to do was wait 5-10 minutes and it would all revert.
Compare this to my cisco/foundry/other experience where I would delay changes until I was in the office (physically colocated with main routers) or calling people to be onsite for what was 99% of the time an innocuous change. The stress of it led to me deferring changes or just skipping them entirely which led to more issues/stress/etc.
I'm really not sure there is a single software feature which improved my life as much as "commit confirmed"
Re: WAN router IP address change blamed for global Microsoft 365 outage
#35> As part of a planned change to update the IP address on a WAN router, a command given to the router caused it to send messages to all other routers in the WAN, which resulted in all of them recomputing their adjacency and forwarding tables. During this re-computation process, the routers were unable to correctly forward packets traversing them. "The command that caused the issue has different behaviors on different…
I feel like they intended to /ADD/ a new loopback IP and in the process accidentally removed the existing one and replaced because I think anyone intentionally changing the loopback IP knows it's going to reset all bgp sessions. I think more modern cisco/arista platforms now "secondary" by default and perhaps that is what bit them?
Re: WAN router IP address change blamed for global Microsoft 365 outage
#36> As part of a planned change to update the IP address on a WAN router, a command given to the router caused it to send messages to all other routers in the WAN, which resulted in all of them recomputing their adjacency and forwarding tables. During this re-computation process, the routers were unable to correctly forward packets traversing them. "The command that caused the issue has different behaviors on different…
This was a rather interesting event. In general, changing the IP address (even the loopback address) shouldn't have caused it from the BGP perspective. For example, if you were to change the IP address of BGP enabled router that has multiple BGP sessions, all other routers tore down the sessions to it, and withdrew the prefixes. BGP reconverge events take time. However, less than this took (90+ minutes and then a few more hours until __full__ recovery).
This seems like one of the events in which they changed IP on Route Reflector routers that were pretty busy, which would cause reconvergence and CPU spikes for all routers that it had sessions with. Also, there was a lot of volatility, as part of which re-advertisements were happening continuously. They also attempted rollback, which caused reverse operation, which triggered reconvergence. The other scenario is doing this change on the SDN controller, which affected all other routers.
More details: https://www.thousandeyes.com/blog/microsoft-outage-analysis-... https://www.thousandeyes.com/resources/na-microsoft-outage-a...
Re: WAN router IP address change blamed for global Microsoft 365 outage
#37Earlier quoted context omitted.
IME Network engineers put too much faith in vendors. They think "the vendor says this is a resilient virtual chassis so it can't break", rather than thinking "ok, if this breaks what happens" A crash affecting both sides of a "resilient" virtual chassis I had to work with took off a major broadcast last year (it was a last minute favour I was doing, and I rerouted to a tertiary route in a couple of minutes). Meanwhil…
This may be true of enterprise network engineers but I’ve worked across a lot of very large networks (telco, not cloud) and we never ever trust the vendor. The kind of bugs that I’ve read about in errata notes over the years is wild and truly unpredictable.
Enterprise is definitely different - network guys need multiple customers to develop the vendor skepticism. I used to get into brutal internal fights with network directors over whatever bullshit the Cisco salesman said offhand that was treated as though it was delivered by Moses off the mountain. One guy tried to get me fired because I offended an SE. lol.
I worked on systems and platforms at the time, and we were more cynical even about vendors we liked.
Re: WAN router IP address change blamed for global Microsoft 365 outage
#38It’s not DNS
There’s no way it’s DNS
It was DNS
Re: WAN router IP address change blamed for global Microsoft 365 outage
#39Earlier quoted context omitted.
It wouldn't be the first time that your redundant vendors end up sharing a conduit for a bunch of fiber somewhere. Guess where that backhoe will start digging?
Redundant vendors in the GP’s context referred to using multiple router vendors, eg Cisco and Juniper. Using multiple connectivity vendors doesn’t guarantee path diversity. Demanding fibre maps and ensuring that your connectivity has separate points of entry into the building, doesn’t cross outside the building, and validating with your DC provider that your cross connects aren’t crossing either, guaranteed path dive…
The GP was clearly talking about whole networks, not just the hardware vendors, if I read that different than the GP intended I'll wait for their correction.
One of the problems that I've seen in practice that with the degree of virtualization at play that it has at the same time become much more easy to in principle be guaranteed 100% independence and in practice it has become much harder to verify that this is the case because of all of the abstraction layers underneath the topology. One of my customers specializes in software that allows one to make such guarantees and this is a non-trivial problem, to put it mildly, especially when the situation becomes more dynamic due to outages from various causes.
Re: WAN router IP address change blamed for global Microsoft 365 outage
#40Earlier quoted context omitted.
Holy shit I have been there and it sucks. I wasn't the guy who made the change, but I was on the long call that followed.
Time to share one of my favorite talks (and speakers) ever - "Debugging Under Fire: Keep your Head when Systems have Lost their Mind" (Bryan Cantrill, GOTO 2017) https://www.youtube.com/watch?v=30jNsCVLpAE
This was an awesome lunch listen, thank you for sharing!