Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

WAN router IP address change blamed for global Microsoft 365 outage

theregister.com

Post reply on HN
Print view View original

41–50 of 68 posts

Re: WAN router IP address change blamed for global Microsoft 365 outage

#41

Post by bogomipz »

wmf wrote: post #27
jacquesm wrote: post #25

Earlier quoted context omitted.

The network is a single point of failure, even if the network itself is redundant!

One possible way to fix that is to replace the network with multiple independent networks. It's really expensive though.

This doesn't really make sense. The modern WAN operates on multiple independent networks - SD-WANs, multiple transit providers, fiber-ring MPLS, EVPN etc. If you propagate a bad network change throughout your autonomous system or backbone you can still have an outage on your hands.

Re: WAN router IP address change blamed for global Microsoft 365 outage

#42

Post by Cyph0n »

jsz0 wrote: post #7
iso1631 wrote: post #4

Earlier quoted context omitted.

That's the norm in network ops. Automated testing is pretty much impossible, easy rollback may be possible depending on exactly what was screwed, but not always. Take this for example, looks like the problem was an unplanned recalculation of routing tables. That's not going to be the case on a small scale test network, and rolling back won't help, indeed in this case it likely would cause more problems.

One of the reasons I got out of network engineering was how frequently the work I was required to do would cause unintended consequences. You can do all your due diligence, get your work blessed by vendor support, and still get blown up by a bug or undocumented behaviors on a regular basis. The conspiratorial part of my brain says these network device makers intentionally provide unreliable software and terrible docu…

> The conspiratorial part of my brain says these network device makers intentionally provide unreliable software and terrible documentation to bolster their support contract profits.

As a dev who has worked at one of the major networking vendors, I can assure you that is the not the case. You’d be surprised by how major bugs are handled internally, especially if the bug affects “important” customers.

Re: WAN router IP address change blamed for global Microsoft 365 outage

#45

Post by wmf »

bogomipz wrote: post #41
wmf wrote: post #27

Earlier quoted context omitted.

One possible way to fix that is to replace the network with multiple independent networks. It's really expensive though.

This doesn't really make sense. The modern WAN operates on multiple independent networks - SD-WANs, multiple transit providers, fiber-ring MPLS, EVPN etc. If you propagate a bad network change throughout your autonomous system or backbone you can still have an outage on your hands.

My point is that you could apply the same principle internally; have two backbones managed by separate teams instead of one.

Re: WAN router IP address change blamed for global Microsoft 365 outage

#46

Post by feyman_r »

nzgrover wrote: post #38

It’s not DNS There’s no way it’s DNS It was DNS Credit: https://www.cyberciti.biz/humour/a-haiku-about-dns/

This reminded me of a talk at SREcon this year https://www.usenix.org/conference/srecon23americas/presentat...

Re: WAN router IP address change blamed for global Microsoft 365 outage

#47

Post by noorkersz »

wmf wrote: post #27
jacquesm wrote: post #25

Earlier quoted context omitted.

The network is a single point of failure, even if the network itself is redundant!

One possible way to fix that is to replace the network with multiple independent networks. It's really expensive though.

yes, instead of one network, many independent networks which then can get connected together, forming a network of networks, some kind of inter-network!

..oh wait. see what I did? ahhAHAHA

Re: WAN router IP address change blamed for global Microsoft 365 outage

#48

Post by noorkersz »

my own WAN IP got changed a few months after my ISP was eating(err, bought) by another larger ISP... now it's a private IPv4 address. I'm pretty sure my 'symmetrical bandwidth' is now only really true when testing it, a technique first invented by Herr Volkswer aus Deutsch-Wagen.

Re: WAN router IP address change blamed for global Microsoft 365 outage

#49

Post by bogomipz »

wmf wrote: post #45
bogomipz wrote: post #41

Earlier quoted context omitted.

This doesn't really make sense. The modern WAN operates on multiple independent networks - SD-WANs, multiple transit providers, fiber-ring MPLS, EVPN etc. If you propagate a bad network change throughout your autonomous system or backbone you can still have an outage on your hands.

My point is that you could apply the same principle internally; have two backbones managed by separate teams instead of one.

That still doesn't make sense though. In the context of a WAN, a backbone is an external network. It routes between your POPs. At any rate, the margin of error and complexity in having two separate backbones networks managed by two separate teams would likely result in more network issues not less. The whole point in having an AS is having a coherent routing policy.
Post reply on HN