Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cellular outage in U.S. hits AT&T, T-Mobile and Verizon users

cnbc.com

Post reply on HN
Print view View original

211–220 of 486 posts

Re: Cellular outage in U.S. hits AT&T, T-Mobile and Verizon users

#211

Post by nathanyz »

PietdeVries wrote: post #178
nathanyz wrote: post #33

Earlier quoted context omitted.

Some of our staff are reporting similar where their partner's phone has service and their's doesn't. Both on same AT&T family plan. So the radio bands may play into it although I would think with latest iPhones, they can use any of the bands from AT&T although I could be wrong.

Can you check if things improve if you turn off 5G and move to 4G/LTE instead?

Good thought, but switching to LTE only didn't work. Same result of ending up in SOS only. Cellular over wifi works perfectly fine though. Wish we could count on better post mortems from the phone companies, but I'm not holding my breath for it.

Re: Cellular outage in U.S. hits AT&T, T-Mobile and Verizon users

#212

Post by jrochkind1 »

wingspar wrote: post #179

I’m on Cricket, which is a prepaid cell plan ‘subsidiary’ of AT&T, and calls/internet are work fine. So not a network outage, but account related?

My Cricket phone is not fine. Android phone, giving me regularly repeated alert messages "No voice service. Temporarily turned off by your carrier."

I think your experience is just that it's not 100% outages, some people still have service.

Re: Cellular outage in U.S. hits AT&T, T-Mobile and Verizon users

#213

Post by Scoundreller »

kevin_nisbet wrote: post #200

To everyone trying to speculate on the root cause, I haven't seen enough information in any of the comments to really draw any conclusions. Having worked on several nationwide cellular issues in Canada when I worked in telecom, we saw nationwide impacts based on any number of causes. - A new route injected in the network caused the routing engines on a type of cellular specific equipment to crash nationwide. This too…

Rogers, of course, blamed their vendor (Ericsson I believe it was). Rogers can do no wrong!

Of course, was fun to see yet another huge org have no back-out/failure plan for their potential enterprise-breaking changes. No/limited IT 101 stuff here.

The only positive thing we learned was that the big 3 (really 2) telcos thought it would be a good idea to give eachother emergency backup sims for the other network to key employees in case their network went down. They did that in 2015, but better late than never.

Fun that Rogers used the same core for wireless and wired connections, so many of us were in total blackout, even if we used a 3rd party internet provider that ran over Rogers. Like, everything including their website was down, corp circuits, everything with non-existent comms from Rogers.

Thankfully my org was multi-homed and switched over its circuits at 6am so on-site mostly continued without issue.

Also fun where the towers remained just powered on enough for phones to stick to them but not be able to do anything, so 9-1-1 calls would just fail, instead of failing-over to other networks. Seems like a deficiency in the GSM spec (or Rogers SIM programming?) that I don’t think was actioned on.

https://en.m.wikipedia.org/wiki/2022_Rogers_Communications_o...

Re: Cellular outage in U.S. hits AT&T, T-Mobile and Verizon users

#214

Post by efitz »

mistrial9 wrote: post #158
efitz wrote: post #35

Reporting status externally is hard. I worked at a cloud provider and while we had very detailed metrics about how our systems were functioning, it was difficult to distill that information in a way that customers would understand if they were being impacted or not, and what the impact would be. Just reporting raw numbers wouldn’t give customers the context to understand what was actually going on. How do you actuall…

because you define thresholds for event classification.. the difference between 10024 customers having failure and 10100 customers having failure is not the question, right? when many hundreds of thousands of customers are failing at once, is that really very difficult to determine? secondly, there are financial, management and information security pressures to NOT REPORT reality to the public. This happens VERY OFTE…

In a cloud, massive outages are the rare events, and technically easy to report.

Small outages happen all the time, and are difficult to report accurately.

I think AWS has pivoted to trying to report status in each individual customer's support portal, so that they can give a dashboard that reports the state of the cloud from that customer's perspective. That way a rack down that only affects a few customers is only reported to those customers, and the dashboard doesn't have to always be red for everyone (or green for everyone, even those affected).

Re: Cellular outage in U.S. hits AT&T, T-Mobile and Verizon users

#215

Post by nathanyz »

Latest AT&T Statement: “Our network teams took immediate action and so far three-quarters of our network has been restored,” the company said. “We are working as quickly as possible to restore service to remaining customers.”

Still down for me though.

Re: Cellular outage in U.S. hits AT&T, T-Mobile and Verizon users

#216

Post by MichaelZuo »

Scoundreller wrote: post #213
kevin_nisbet wrote: post #200

To everyone trying to speculate on the root cause, I haven't seen enough information in any of the comments to really draw any conclusions. Having worked on several nationwide cellular issues in Canada when I worked in telecom, we saw nationwide impacts based on any number of causes. - A new route injected in the network caused the routing engines on a type of cellular specific equipment to crash nationwide. This too…

Rogers, of course, blamed their vendor (Ericsson I believe it was). Rogers can do no wrong! Of course, was fun to see yet another huge org have no back-out/failure plan for their potential enterprise-breaking changes. No/limited IT 101 stuff here. The only positive thing we learned was that the big 3 (really 2) telcos thought it would be a good idea to give eachother emergency backup sims for the other network to key…

> Fun that Rogers used the same core for wireless and wired connections, so many of us were in total blackout, even if we used a 3rd party internet provider that ran over Rogers.

If it ran over Rogers circuits then why wouldn't it go down too? Isn't that the case everywhere?

Re: Cellular outage in U.S. hits AT&T, T-Mobile and Verizon users

#218

Post by IAmGraydon »

nathanyz wrote: post #215

Latest AT&T Statement: “Our network teams took immediate action and so far three-quarters of our network has been restored,” the company said. “We are working as quickly as possible to restore service to remaining customers.” Still down for me though.

Still down for me too.

Re: Cellular outage in U.S. hits AT&T, T-Mobile and Verizon users

#219

Post by kevin_nisbet »

Scoundreller wrote: post #213
kevin_nisbet wrote: post #200

To everyone trying to speculate on the root cause, I haven't seen enough information in any of the comments to really draw any conclusions. Having worked on several nationwide cellular issues in Canada when I worked in telecom, we saw nationwide impacts based on any number of causes. - A new route injected in the network caused the routing engines on a type of cellular specific equipment to crash nationwide. This too…

Rogers, of course, blamed their vendor (Ericsson I believe it was). Rogers can do no wrong! Of course, was fun to see yet another huge org have no back-out/failure plan for their potential enterprise-breaking changes. No/limited IT 101 stuff here. The only positive thing we learned was that the big 3 (really 2) telcos thought it would be a good idea to give eachother emergency backup sims for the other network to key…

> Also fun where the towers remained just powered on enough for phones to stick to them but not be able to do anything, so 9-1-1 calls would just fail, instead of failing-over to other networks. Seems like a deficiency in the GSM spec (or Rogers SIM programming?) that I don’t think was actioned on.

Actually, I think this is going to change after the Rogers outage, it's just slowly happening behind the scenes so it's not getting much attention these days. The government has mandated a lot of industry response to failover between providers... we'll see where they land after all the lobbying happens. I do think implementations are changing a bit around this, mostly in the phones so that they give up and go into a network scan if the emergency call is failing.

I worked mostly on core network stuff, so I was a layer removed from the towers, but if they hadn't lost management access they would've been able to tell the tower to stop advertising the network and 911 service. I do understand the question of from a vendor implementation perspective of how automatic this should be though... because automation in this regard does have some of it's own risks and could complicate some types of outages or inadvertently trigger and confuse recovery of problems.

I'm with you though there should be an automatic mechanism to fail over to other network operators, I just haven't thought through all the risks with it and I hope the industry is taking their time to think through the implications.

Post reply on HN