Live data from Hacker News

UK: Phone networks down: EE, BT, Three, Vodafone, O2 not working in mass outage

the-independent.com

11–20 of 111 posts

Re: UK: Phone networks down: EE, BT, Three, Vodafone, O2 not working in mass outage

#11
post #3

Another Down Detector bullshit article.... it's getting incredibly tiring. Every time a provider (Phone, Internet or even cloud services) suffer issues ALL of them are reported as down.

Who downs detector the down detector? Or even better who is the redundancy for dd if the site is actually up?

Re: UK: Phone networks down: EE, BT, Three, Vodafone, O2 not working in mass outage

#12
post #4

I wonder if it will be the same cause as the big Rogers outage in '22, a good old fashioned BGP botch. https://www.reuters.com/business/media-telecom/rogers-commun... https://en.wikipedia.org/wiki/2022_Rogers_Communications_out...

Diff with Rogers is that they took out their entire network: cellular, home/biz internet, home phone, corporate circuits (including MPLS links), most cable TV, a bunch of their broadcast radio (AM/FM) network just dead dead dead.

Well, their towers were sorta up (as they couldn’t remotely turn them off since the network was down), so if you had a Rogers SIM, a call to 9-1-1 wouldn’t failover to other networks because the device made just enough of a handshake to try and fail on the Rogers network. A flaw in GSM I reckon.

Apparently the workaround was to remove/disable your SIM and hope another network has a stronger signal.

Oh, and the CTO was on holiday and had no idea for a while because… their phone was on roaming with Rogers and therefore dead.

I wonder if Rogers still does planned-in-advance multi-stage potentially-enterprise-breaking updates on Fridays

Re: UK: Phone networks down: EE, BT, Three, Vodafone, O2 not working in mass outage

#14
post #2

I see we’re still using Down Detector as a source for stories, which may as well be called People Are Talking on Twitter Detector. It doesn’t do anything smart, it’s just looking for keywords on social media, sometimes by coincidence this indicates an actual outage.

Do you have a better suggestion? This is the best we can get, without having a public protocol for asking any service, "are you alive?"

Re: UK: Phone networks down: EE, BT, Three, Vodafone, O2 not working in mass outage

#16
post #4

I wonder if it will be the same cause as the big Rogers outage in '22, a good old fashioned BGP botch. https://www.reuters.com/business/media-telecom/rogers-commun... https://en.wikipedia.org/wiki/2022_Rogers_Communications_out...

Diff with Rogers is that they took out their entire network: cellular, home/biz internet, home phone, corporate circuits (including MPLS links), most cable TV, a bunch of their broadcast radio (AM/FM) network just dead dead dead. Well, their towers were sorta up (as they couldn’t remotely turn them off since the network was down), so if you had a Rogers SIM, a call to 9-1-1 wouldn’t failover to other networks because…

In a financial company I worked at we would do some of the biggest, riskiest changes at 5pm on a Friday (or Saturday evening if we were worried about impacting international trades). The logic being that we would have the most time to fix things before markets open monday.

Re: UK: Phone networks down: EE, BT, Three, Vodafone, O2 not working in mass outage

#19
post #6
post #2

I see we’re still using Down Detector as a source for stories, which may as well be called People Are Talking on Twitter Detector. It doesn’t do anything smart, it’s just looking for keywords on social media, sometimes by coincidence this indicates an actual outage.

Yeah, it's far from ideal, but in my experience its accuracy is better than most anything else readily available, including the official status pages maintained by most tech companies.

Yeah, and not only do you get to see if it's down or not (reddit infamously always says it's up even when there are issues), but you also get to see the raw data of reports. Ofttimes I've seen the trend go up and realized it's a very recent issue - even before downdetector itself recognizes it as such.

Human reading > DD reading >> "All our services are operational" when they're absolutely f--ing not.

Re: UK: Phone networks down: EE, BT, Three, Vodafone, O2 not working in mass outage

#20
post #2

I see we’re still using Down Detector as a source for stories, which may as well be called People Are Talking on Twitter Detector. It doesn’t do anything smart, it’s just looking for keywords on social media, sometimes by coincidence this indicates an actual outage.

Actually it’s quite smart! Probably the smartest possible solution to the problem they tried to solve.

“Smart” doesn’t have to mean complex and technically sophisticated.

Post reply on HN