Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

WAN router IP address change blamed for global Microsoft 365 outage

theregister.com

Post reply on HN
Print view View original

21–30 of 68 posts

Re: WAN router IP address change blamed for global Microsoft 365 outage

#21

Post by throw0101a »

iso1631 wrote: post #4
alex-mohr wrote: post #3

Earlier quoted context omitted.

It does seem like network configuration remains rather manual compared to other large scale systems that include more automation. In Microsoft's case, the remediation is not to put in place higher level systems to safely accomplish the goal of the command. Instead: - "We have blocked highly impactful commands from getting executed on the devices (Completed)" - "We will require all command execution on the devices to…

That's the norm in network ops. Automated testing is pretty much impossible, easy rollback may be possible depending on exactly what was screwed, but not always. Take this for example, looks like the problem was an unplanned recalculation of routing tables. That's not going to be the case on a small scale test network, and rolling back won't help, indeed in this case it likely would cause more problems.

> That's the norm in network ops. Automated testing is pretty much impossible, easy rollback may be possible depending on exactly what was screwed, but not always.

Ansible/Napalm is a thing in NetOps in some places. Some folks use Eve-ng / GNS3 to spin up virtual networks to test config changes, and it may be possible to do CI/CD changes if you track things in a repo.

Juniper JunOS has auto-rollback if you don't confirm the change after "x" minutes:

* https://www.juniper.net/documentation/us/en/software/junos/c...

So if you did something that causes breakage and disconnection from the router, you (ideally) don't have to do anything but wait it out.

Re: WAN router IP address change blamed for global Microsoft 365 outage

#23

Post by jgrahamc »

newah1 wrote: post #19
jgrahamc wrote: post #2

Having, uh, had bad things happen with router configuration I feel for them. https://blog.cloudflare.com/cloudflare-outage-on-july-17-202...

I remember this happening. The 20 some sites we ran went down as they were supported by cloudflare. I spent a panicked 30 minutes trying to figure out what I had done wrong, to eventually find out it was on CF's end. I remember voicing at our team meeting "boy, they must be panicking at CloudFlare." Cloudflare works so spectacularly we just wrote it off as a one time thing.

There was no panic but there was a lot of VUF (Very Urgent Focus)!

Re: WAN router IP address change blamed for global Microsoft 365 outage

#25

Post by jacquesm »

jimmyl02 wrote: post #9

It seems that in modern large scale systems networking continues to be one of the few things were a a seemingly small and inconsequential change can cause entire cloud providers and highly redundant systems to go down. It makes sense as networking is the fabric connecting all systems together but each time an incident like this occurs I'm reminded of just how important networking is. Network engineers and the people…

The network is a single point of failure, even if the network itself is redundant!

Re: WAN router IP address change blamed for global Microsoft 365 outage

#26

Post by jacquesm »

ccakes wrote: post #16
iso1631 wrote: post #11

Earlier quoted context omitted.

IME Network engineers put too much faith in vendors. They think "the vendor says this is a resilient virtual chassis so it can't break", rather than thinking "ok, if this breaks what happens" A crash affecting both sides of a "resilient" virtual chassis I had to work with took off a major broadcast last year (it was a last minute favour I was doing, and I rerouted to a tertiary route in a couple of minutes). Meanwhil…

This may be true of enterprise network engineers but I’ve worked across a lot of very large networks (telco, not cloud) and we never ever trust the vendor. The kind of bugs that I’ve read about in errata notes over the years is wild and truly unpredictable.

It wouldn't be the first time that your redundant vendors end up sharing a conduit for a bunch of fiber somewhere. Guess where that backhoe will start digging?

Re: WAN router IP address change blamed for global Microsoft 365 outage

#27

Post by wmf »

jacquesm wrote: post #25
jimmyl02 wrote: post #9

It seems that in modern large scale systems networking continues to be one of the few things were a a seemingly small and inconsequential change can cause entire cloud providers and highly redundant systems to go down. It makes sense as networking is the fabric connecting all systems together but each time an incident like this occurs I'm reminded of just how important networking is. Network engineers and the people…

The network is a single point of failure, even if the network itself is redundant!

One possible way to fix that is to replace the network with multiple independent networks. It's really expensive though.

Re: WAN router IP address change blamed for global Microsoft 365 outage

#28

Post by candiddevmike »

jsz0 wrote: post #7
iso1631 wrote: post #4

Earlier quoted context omitted.

That's the norm in network ops. Automated testing is pretty much impossible, easy rollback may be possible depending on exactly what was screwed, but not always. Take this for example, looks like the problem was an unplanned recalculation of routing tables. That's not going to be the case on a small scale test network, and rolling back won't help, indeed in this case it likely would cause more problems.

One of the reasons I got out of network engineering was how frequently the work I was required to do would cause unintended consequences. You can do all your due diligence, get your work blessed by vendor support, and still get blown up by a bug or undocumented behaviors on a regular basis. The conspiratorial part of my brain says these network device makers intentionally provide unreliable software and terrible docu…

Networking and storage changes are always butt clenching affairs. Way more stressful than anything else in IT due to their blast radius if something shits the bed.

Re: WAN router IP address change blamed for global Microsoft 365 outage

#29

Post by jacquesm »

wmf wrote: post #27
jacquesm wrote: post #25

Earlier quoted context omitted.

The network is a single point of failure, even if the network itself is redundant!

One possible way to fix that is to replace the network with multiple independent networks. It's really expensive though.

Yes, exactly. Most really mission critical places do exactly that.

The first time I saw something like that put into practice was when an experiment in the oil and gas industry that was scheduled to run for years delivered their network design. On the runtime cost of the experiment the extra network wasn't a big deal, but a service interruption would have been and would have caused them to have to restart the whole thing from scratch. It's more than a decade ago and I forgot what the exact context was but the whole thing was fascinating from a redundancy perspective as well as the degree of thinking that had gone into the risk assessment. Those guys really knew their business. Also the amount of data that experiment was expected to generated was off the scale. Multiple petabytes, which at the time (a decade ago or so) was a non trivial amount of data.

Re: WAN router IP address change blamed for global Microsoft 365 outage

#30

Post by candiddevmike »

septune wrote: post #24

I miss the old days of IOS : switchport trunk allowed vlan (add) xxx Can’t imagine how many outages where caused by the missing « add » command.

Too many Cisco commands would truncate the syntax if you didnt know better:

no access-list 101 permit something

so long access-list 101!

Post reply on HN