Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

WAN router IP address change blamed for global Microsoft 365 outage

theregister.com

Post reply on HN
Print view View original

61–68 of 68 posts

Re: WAN router IP address change blamed for global Microsoft 365 outage

#61

Post by iso1631 »

AdamJacobMuller wrote: post #34
throw0101a wrote: post #21

Earlier quoted context omitted.

> That's the norm in network ops. Automated testing is pretty much impossible, easy rollback may be possible depending on exactly what was screwed, but not always. Ansible/Napalm is a thing in NetOps in some places. Some folks use Eve-ng / GNS3 to spin up virtual networks to test config changes, and it may be possible to do CI/CD changes if you track things in a repo. Juniper JunOS has auto-rollback if you don't conf…

commit confirmed is such a life-saver. I ran a production network which spanned multiple continents and even though I probably only ever actually needed commit confirmed a single digit number of times, the fact that it was there made every change I did 99% less stressful. I knew that even if I made a mistake, all I had to do was wait 5-10 minutes and it would all revert. Compare this to my cisco/foundry/other experie…

So instead of one ripple across your BGP network, you have two as it rollsback the change?

The problem is that the state in routing tables isn't stored in a single location, it's dynamically built over time. Breaking a single router in the wrong way can break the state, and there's no rollback of that state

Re: WAN router IP address change blamed for global Microsoft 365 outage

#62

Post by iso1631 »

throw0101a wrote: post #21
iso1631 wrote: post #4

Earlier quoted context omitted.

That's the norm in network ops. Automated testing is pretty much impossible, easy rollback may be possible depending on exactly what was screwed, but not always. Take this for example, looks like the problem was an unplanned recalculation of routing tables. That's not going to be the case on a small scale test network, and rolling back won't help, indeed in this case it likely would cause more problems.

> That's the norm in network ops. Automated testing is pretty much impossible, easy rollback may be possible depending on exactly what was screwed, but not always. Ansible/Napalm is a thing in NetOps in some places. Some folks use Eve-ng / GNS3 to spin up virtual networks to test config changes, and it may be possible to do CI/CD changes if you track things in a repo. Juniper JunOS has auto-rollback if you don't conf…

Emulating even a mid-sized network in GNS3 requires massive resources, and my cisco account manager doesn't seem to even get why I'd want to deploy a test system of 50 different multi-vendor switches (and key supporting services like syslog and tacacs) with terraform, run some tests, apply a configuration change, and run more tests.

And virtual switches aren't the same as physical switches in any case, they have different bugs, different features, different responsiveness.

Re: WAN router IP address change blamed for global Microsoft 365 outage

#63

Post by iso1631 »

oarsinsync wrote: post #32
jacquesm wrote: post #26

Earlier quoted context omitted.

It wouldn't be the first time that your redundant vendors end up sharing a conduit for a bunch of fiber somewhere. Guess where that backhoe will start digging?

Redundant vendors in the GP’s context referred to using multiple router vendors, eg Cisco and Juniper. Using multiple connectivity vendors doesn’t guarantee path diversity. Demanding fibre maps and ensuring that your connectivity has separate points of entry into the building, doesn’t cross outside the building, and validating with your DC provider that your cross connects aren’t crossing either, guaranteed path dive…

Its a bit of both. Internationally I find I can't trust the network maps of the connectivity vendors and I'm better going for two separate companies (ones which are part of different subsea cables -- e.g. Wiocc on Eassy and Safaricom on TEAMS).

Of course I had one failure in Delhi which the provider blamed on 5 separate fibre cuts. Long distance circuits can run via areas where they can sustain multiple cuts across large amounts of area (regional flooding is a good one), and fixing isn't instant. This can be mittigated a little, but you still end up with circuit issues -- I had two fibre runs into Shetland the other month. Frist one was cut, c'est la vie. Second one was cut, had to use a very limited RF link. There's only so much you can do.

On the other hand I've just been given a BT Openreach plan which lists any pinch points of a new RO2 EAD install, I can see the closest the two get during transport is about 400m (aside from the end point of course, and experience has taught me I can trust it.

Re: WAN router IP address change blamed for global Microsoft 365 outage

#64

Post by bogomipz »

wmf wrote: post #56
bogomipz wrote: post #49

Earlier quoted context omitted.

That still doesn't make sense though. In the context of a WAN, a backbone is an external network. It routes between your POPs. At any rate, the margin of error and complexity in having two separate backbones networks managed by two separate teams would likely result in more network issues not less. The whole point in having an AS is having a coherent routing policy.

You're talking like multihoming doesn't work. Sure there are cases where bugs or bad configs can propagate across ASes but most of the time you can survive if one provider goes down.

And that's exactly where the whole "have two backbones managed by separate teams instead of one" stops. If someone pushes out an incorrect network config to the end box then all that "let's have two of everything" becomes completely worthless. And as far as multihoming everything and having every single box on the network act as router, unless you are running a CDN of some sort, really makes zero sense. You seem to be arguing that adding more complexity will automatically result in better reliability.

Re: WAN router IP address change blamed for global Microsoft 365 outage

#65

Post by allarm »

admax88qqq wrote: post #57
bogomipz wrote: post #55

Earlier quoted context omitted.

The parent was stating that two networks would be better but its none done because of costs. And that's complete nonsense. The fact that it's more difficult and complex to have two separate teams manage two separate networks means it's more prone to error and misconfiguration. The reason it's not done has nothing do with financial costs but rather because it makes no sense, for the very fact I just mentioned.

Two end to end networks would be more reliable. Like two independent internets spanning from your server to my laptop. Two completely isolated end to end transports. That's what OP meant when they said you could make it more reliable by having a redundant network. It's just prohibitively expensive. Then if one internet goes down in any way I talk to you over the other. That's a fairly straightforward fallback algorit…

Actually I have seen a setup that was quite close to this. Two separate networks, one of them was completely isolated from another, didn’t have Internet access and used a separate set of network equipment. On top of that, the building itself had two entrances - one for the boss and another one for the personnel. You physically couldn’t get from one part of the building to another. It didn’t help the boss though - he was blown up in his car one day. Fun times.

Re: WAN router IP address change blamed for global Microsoft 365 outage

#66

Post by allarm »

bogomipz wrote: post #33
spookthesunset wrote: post #12

Earlier quoted context omitted.

I remember the first time I got access to an employers production Cisco router. It’s pretty scary how easy it is to majorly fuck something up. There isn’t a concept of a transaction or a rollback. You just enter a command, press enter and it’s live. To counter this we’d write all the commands we planned on executing and peer review it. Nothing was to be done “on the fly” (at least in theory) In short, coming from a d…

>"There isn’t a concept of a transaction or a rollback. You just enter a command, press enter and it’s live." This hasn't been true for a very long time. Juniper router's have rollbacks, commits and revisions: https://www.juniper.net/documentation/us/en/software/junos/c... and https://www.juniper.net/documentation/us/en/software/junos/c... Cisco has similar: https://www.cisco.com/c/en/us/td/docs/ios/ios_xe/fundamenta…

Except Cisco doesn’t have a commit feature in any of their OS and the rollback feature is not implemented everywhere as well - NXOs doesn’t have it for example. Still, it’s better than ‘reload in 5’ that we had to use back then.

Re: WAN router IP address change blamed for global Microsoft 365 outage

#67

Post by oarsinsync »

jacquesm wrote: post #39
oarsinsync wrote: post #32

Earlier quoted context omitted.

Redundant vendors in the GP’s context referred to using multiple router vendors, eg Cisco and Juniper. Using multiple connectivity vendors doesn’t guarantee path diversity. Demanding fibre maps and ensuring that your connectivity has separate points of entry into the building, doesn’t cross outside the building, and validating with your DC provider that your cross connects aren’t crossing either, guaranteed path dive…

The GP was clearly talking about whole networks, not just the hardware vendors, if I read that different than the GP intended I'll wait for their correction. One of the problems that I've seen in practice that with the degree of virtualization at play that it has at the same time become much more easy to in principle be guaranteed 100% independence and in practice it has become much harder to verify that this is the…

> One of the problems that I've seen in practice that with the degree of virtualization at play

If you’re buying SDN WAN solutions, you get what you get.

If you’re buying specific paths, you get what you pay for.

Re: WAN router IP address change blamed for global Microsoft 365 outage

#68

Post by AdamJacobMuller »

iso1631 wrote: post #61
AdamJacobMuller wrote: post #34

Earlier quoted context omitted.

commit confirmed is such a life-saver. I ran a production network which spanned multiple continents and even though I probably only ever actually needed commit confirmed a single digit number of times, the fact that it was there made every change I did 99% less stressful. I knew that even if I made a mistake, all I had to do was wait 5-10 minutes and it would all revert. Compare this to my cisco/foundry/other experie…

So instead of one ripple across your BGP network, you have two as it rollsback the change? The problem is that the state in routing tables isn't stored in a single location, it's dynamically built over time. Breaking a single router in the wrong way can break the state, and there's no rollback of that state

> So instead of one ripple across your BGP network, you have two as it rollsback the change?

It's possible, it depends on what the nature of the change is. If you use super short commit confirmed intervals (commit confirmed 1) then yes you can cause a situation where you revert a "good" commit and cause a second disturbance. You need to intelligently reason about commit confirmed times to consider this when you're making such changes.

Post reply on HN