O2 outage: 31m mobile customers unable to get online • Hacker News Board

Re: O2 outage: 31m mobile customers unable to get online

#11

Post by Angostura » Thu, Dec 06, 2018, 4:09 PM UTC

otoburb wrote: post #9

bArray wrote: post #7

Earlier quoted context omitted.

Why do so many status pages seem to crash/incorrect report when the service goes down? Once you get on the page: We're really sorry but this system is currently experiencing some technical problems. Our engineers have been notified, so please check back later.

Best practice is to host the status page on completely separate infrastructure. This adds a lot of operational overhead often seen as disproportionately low benefit relative to the ongoing effort to maintain, until rare events like this actually happen.

> Best practice is to host the status page on completely separate infrastructure.

Perhaps they have - and it's overloaded.

Re: O2 outage: 31m mobile customers unable to get online

#12

Post by betaby » Thu, Dec 06, 2018, 4:12 PM UTC

dreen wrote: post #2

https://status.o2.co.uk/ > We believe other mobile operators around the world are also affected. Anyone else affected?

Statistically speaking - yes. I realy want to know what O2 had in mind by saying that.

Re: O2 outage: 31m mobile customers unable to get online

#13

Post by verytrivial » Thu, Dec 06, 2018, 4:23 PM UTC

I'm seeing references to something Ericsson contributes to their network. It would very interesting to see what went wrong and in particular, what caused the change or deployment to be "un-revertable". My money is on a cascade of issues, some of which were latent and only triggered today.

This is, by far, the worst cell outage I've seen in the UK. There is zero connectivity via O2 in e.g. London, data or otherwise.

[1] https://www.standard.co.uk/news/world/o2-network-down-ericss...

Re: O2 outage: 31m mobile customers unable to get online

#14

Post by martinald » Thu, Dec 06, 2018, 4:25 PM UTC

verytrivial wrote: post #13

I'm seeing references to something Ericsson contributes to their network. It would very interesting to see what went wrong and in particular, what caused the change or deployment to be "un-revertable". My money is on a cascade of issues, some of which were latent and only triggered today. This is, by far, the worst cell outage I've seen in the UK. There is zero connectivity via O2 in e.g. London, data or otherwise. […

Due to an expired TLS cert: https://www.ericsson.com/en/press-releases/2018/12/update-on...

Re: O2 outage: 31m mobile customers unable to get online

#15

Post by enlyth » Thu, Dec 06, 2018, 4:45 PM UTC

I haven't had mobile internet since yesterday evening on O2, and I travel to London for work.

Luckily, there's WiFi at home, at work, and on the trains, so it's not really affecting me too much, but I'd really like to see the postmortem of such a huge outage.

Re: O2 outage: 31m mobile customers unable to get online

#16

Post by acupofnope » Thu, Dec 06, 2018, 4:56 PM UTC

betaby wrote: post #12

dreen wrote: post #2

https://status.o2.co.uk/ > We believe other mobile operators around the world are also affected. Anyone else affected?

Statistically speaking - yes. I realy want to know what O2 had in mind by saying that.

The issues are in some of Ericsson network software, and they have other customers apparently.

https://www.ericsson.com/en/press-releases/2018/12/update-on...

Re: O2 outage: 31m mobile customers unable to get online

#17

Post by setquk » Thu, Dec 06, 2018, 5:10 PM UTC

verytrivial wrote: post #13

I'm seeing references to something Ericsson contributes to their network. It would very interesting to see what went wrong and in particular, what caused the change or deployment to be "un-revertable". My money is on a cascade of issues, some of which were latent and only triggered today. This is, by far, the worst cell outage I've seen in the UK. There is zero connectivity via O2 in e.g. London, data or otherwise. […

There was worse on 7/7/2005 in London due to the bombings. Nearly all the providers fell flat due to overload including land lines and mobile phones. Also public transport was hosed. I walked 11 miles home in the end.

FM radio was about the only thing working. Thank you cheap nokia!

I have an iPhone now which is devoid of an FM radio but carry a multiband VHF transceiver with me most of the time now as that can pick up FM and do comms on 2m (I have an amateur radio license as well).

Re: O2 outage: 31m mobile customers unable to get online

#18

Post by boyce » Thu, Dec 06, 2018, 5:34 PM UTC

roliver wrote: post #6

boyce wrote: post #5

I thought initially it was a local thing like when Vodafone disappeared here because a falcon made a nest on the transmitter

University of Southampton?

Yup. 2006-2012, still live in Highfield.

Re: O2 outage: 31m mobile customers unable to get online

#19

Post by aarong11 » Thu, Dec 06, 2018, 6:24 PM UTC

Angostura wrote: post #11

otoburb wrote: post #9

Earlier quoted context omitted.

Best practice is to host the status page on completely separate infrastructure. This adds a lot of operational overhead often seen as disproportionately low benefit relative to the ongoing effort to maintain, until rare events like this actually happen.

> Best practice is to host the status page on completely separate infrastructure. Perhaps they have - and it's overloaded.

Probably offset by the amount of people who can't access it due to their internet not working haha

Re: O2 outage: 31m mobile customers unable to get online

#20

Post by noir_lord » Thu, Dec 06, 2018, 6:34 PM UTC

martinald wrote: post #14

verytrivial wrote: post #13

I'm seeing references to something Ericsson contributes to their network. It would very interesting to see what went wrong and in particular, what caused the change or deployment to be "un-revertable". My money is on a cascade of issues, some of which were latent and only triggered today. This is, by far, the worst cell outage I've seen in the UK. There is zero connectivity via O2 in e.g. London, data or otherwise. […

Due to an expired TLS cert: https://www.ericsson.com/en/press-releases/2018/12/update-on...

I liked the "limited number of customers" yes that's true, it affect 2-3 companies, it's just that a single one of those companies had 32M customers.

Ouch.