191–200 of 486 posts
I'll take a chance here and say this was a hack, possibly at the equipment-level. One major carrier having an outage? Possible. But three? On different networks? Even if its not a hack I'd love to see the root cause on this one! Communications is critical infrastructure, so I'm gonna guess the government will demand a full report.
NOAA is reporting R3 activity. Solar flare outage seems likely
My ATT Phone is in SOS mode. However, ATT's outages status reports: > All clear! No outages to report. > We didn’t find any outages in your area. Still having issues? https://www.att.com/outages/
If you tried to make a 9-1-1 call, it would just fail. It wouldn’t fail over to another network because the towers were still powered up but unable to do anything, and Rogers couldn’t power them down because their internal stuff was all down.
Like a day later they said you could remove your SIM card to do a 9-1-1 call. Thanks guys.
Of course, no real info from the provider during the outage. Turns out they did an enterprise-risking upgrade on a Friday morning and nobody at the org seemed to have a “what if this fails plan”. CTO was on vacation and roaming phones were black too and he thought it was just an issue for him.
https://en.m.wikipedia.org/wiki/2022_Rogers_Communications_o...
NOAA is reporting R3 activity. Solar flare outage seems likely
A solar flare targetted at ATT Infra only? Unlikely. Could be a hack Could be a single point of failure Could be a config change that borked the system Could be other things that make more sense we will have to wait for more info.
I haven't had service since around 3am this morning, my internet (fixed wireless) was also down around that time but has since come back on. I had just assumed it was local tower maintenance until I woke up and see it's still out and it's effecting millions. But was surprised to see it barely mentioned/getting engagement here given the scale.
Earlier quoted context omitted.
On the latest iPhones, SOS mode is the emergency fallback to satellite service. It's really meant to be used in situations where you're well outside of any sort of service area but you have a clear, unobstructed view of the sky. Your iPhone will instruct you on where to point and help you track an emergency satellite that is manned by live humans who will take your emergency request and relay it to the proper people.…
SOS means it has cell service and you can call 911. If there is no cell service then it's SOS with a little picture of a satellite next to it.
My ATT Phone is in SOS mode. However, ATT's outages status reports: > All clear! No outages to report. > We didn’t find any outages in your area. Still having issues? https://www.att.com/outages/
- A new route injected in the network caused the routing engines on a type of cellular specific equipment to crash nationwide. This took down internet access only from cell devices nationwide. But most people didn't notice because it happened at 2AM maintenance window and was fortunately discovered and reversed before business hours why the routing engine was in a crash loop.
- A tech plugged in some new routers, and the existing core routers crashed and rebooted. While the news worthy impact was just a regional outage for something like 20 minutes, we discovered bugs and side effects from the Pacific to Atlantic coasts over the next 12 hours. So when you say you're impacted at location x, that data point could be everyone is down in the area, many people are having issues, or only one or two people have issues spilled over to some other region. This is why seeing it does or doesn't work in location x is limited value, as almost every outage I've investigated could result in some people still having service for various reasons. The question is in a particular area is it 100% impact, 50% impact, or 0.001% impact.
- A messaging relay ran into it's configured rate limit. Retries in the protocol increased the messaging rate, so we effectively had a congestion collapse in that particular protocol. Because this was a congestion issue on passing state around, there were nationwide impacts, but you still had x% chance of completing your message flows and getting service.
And then there was the famous Rogers outage where I don't remember them admitting to the full root cause. It's speculated that they did an upgrade on change on their routing network, which also had the side effect of the problem booting all the technicians from the network. Then recovery was difficult because the issue took out the nationwide network and broke the ability for employees to coordinate (you know because they use the same network as all the customers who also can't get service). All the CRTC filings I reviewed had all the useful information redacted though, so there isn't much we can learn from it.
So it's fun to speculate, but here's hoping at the end of the day ATT is more transparent then we are in Canada, so the rest of the industry can learn from the experience.