261–270 of 486 posts
What's also odd is I'm not even able to log into my ATT.com account.
When I try to log in, it states "want to pay your bill".
So my immediate concern was, did my credit card expire and my service was turned off.
I'm still hoping that's not the case, and it's just that I'm impacted by this outage (and not something else).
I wonder if this is a cyber incident. Curious if any telecom folks know what the most likely explanation for an event like this would be, and what telltale signs/symptoms might first indicate this was caused by something nefarious.
> However, the US Cybersecurity and Infrastructure Security Agency is “working closely with AT&T to understand the cause of the outage and its impacts, and stand[s] ready to offer any assistance needed,” Eric Goldstein, the agency’s executive assistant director for cybersecurity, said in a statement to CNN.[1] [1] - [ https://www.cnn.com/2024/02/22/tech/att-cell-service-outage/... This isn't telling of anything, righ…
> “Everybody’s incentives are aligned,” the former official said. “The FCC is going to want to know what caused it so that lessons can be learned. And if they find malfeasance or bad actions or, just poor quality of oversight of the network, they have the latitude to act.”
If AT&T gets to decide if they are at fault, they will, of course, never be at fault. So a third-party investigation makes a lot of sense.
I would also suspect that the FCC would not be as well versed in determining if there was a hack or even who did it, which is why I feel like CISA would need to get involved in the investigation.
Earlier quoted context omitted.
> Someone in this post commented their military friend says get gas The Rogers outage in Canada took out the nationwide debit card payment network because that infra depended on Rogers. Credit cards still worked, but depends on your station’s access to make the transaction. And no shortage of shops running their POS “in the cloud” and needing to close if they lose internet access. I actually did have to lend cash to…
PoS devices are usually networked. If you don't validate transactions in realtime you would later validate in batch, but that has more risk than validating at the time of transaction.
yeah, a lot of orgs just don't enable that (or don't have a process to enable it as required, and have difficulty pushing out a notice to do so if the network is down!).
Also can only do offline credit card transactions. Can't with our Interac (Canadian-only) debit network. Unsure about Visa/Mastercard debit transactions.
What is the big deal? Just use your HAM radio.
Late yesterday, I tried to place a 911 call while driving on the 101 north toward San Francisco. Before I could complete my report, the call dropped, and for the next 10 or so minutes I had no service, only "SOS". I'm on Verizon, and the timing doesn't match up with this headline, but now I'm suspicious.
Earlier quoted context omitted.
Rogers, of course, blamed their vendor (Ericsson I believe it was). Rogers can do no wrong! Of course, was fun to see yet another huge org have no back-out/failure plan for their potential enterprise-breaking changes. No/limited IT 101 stuff here. The only positive thing we learned was that the big 3 (really 2) telcos thought it would be a good idea to give eachother emergency backup sims for the other network to key…
> Also fun where the towers remained just powered on enough for phones to stick to them but not be able to do anything, so 9-1-1 calls would just fail, instead of failing-over to other networks. Seems like a deficiency in the GSM spec (or Rogers SIM programming?) that I don’t think was actioned on. Actually, I think this is going to change after the Rogers outage, it's just slowly happening behind the scenes so it's…
Earlier quoted context omitted.
> Also fun where the towers remained just powered on enough for phones to stick to them but not be able to do anything, so 9-1-1 calls would just fail, instead of failing-over to other networks. Seems like a deficiency in the GSM spec (or Rogers SIM programming?) that I don’t think was actioned on. Actually, I think this is going to change after the Rogers outage, it's just slowly happening behind the scenes so it's…
> I do think implementations are changing a bit around this, mostly in the phones so that they give up and go into a network scan if the emergency call is failing It seems like this is a global problem, since all Rogers-subscribed devices in a Rogers reception area couldn’t make 9-1-1 calls. But could be a SIM coding issue and not afflict other providers elsewhere. I just always imagined the GSM spec was so resilient…
As I recall it is slightly more nuanced than this and was particular to the failure mode, and has a couple of different things aligning to create the failure mode.
If you're phone is just blank, no sim card. To make an emergency call, it has to just start scanning all the supported frequencies. This is very slow, tune radio, wait for the scheduled information block that described the network on the radio protocol. See if it has the emergency services bit enabled. If not, tune to next frequency and try again. I used to remember all the timers, but almost a decade later I can't remember all the network timers for the information blocks.
The sim card interaction, is say you're at home and you boot up your phone with 100% clean state. You don't want to wait for this scan to complete, so the SIM card gives the phone hints about which frequencies the carrier uses, so start on frequency x to find the network. But if you roam internationally, it can take alot longer to find a partner network, and there are some other techs around steering to preferred partners, but I don't know that those come into play here. I don't know but would be surprised if there is a SIM option to try and pin the emergency calls to a network, I think it's more likely the interaction is this hint on where to start the scan.
The way the rogers network failed, it appears to me it caused the towers to stay in a state where they advertised in their radio block the network was there, and the 911 bit was enabled so the network could be used for emergency calls. This is where I don't really have the details since they haven't been public about it, how much of their network was still available internally. Maybe the cell towers could all see each other, that network layer was OK, and the signalling equipment was all talking to each other as well. That's the part I don't really know and have to speculate, as well as the tower side since I was a core person. So because the towers had enough service to never wilt themselves, they kept advertising the network, along with the 911 support. But then when you try to activate an emergency call, somewhere in the signalling path, as you get from tower to signalling system, to the voip equipment, to the circuits to the emergency center the outage knocked something out. Oh and for all these pieces of 911 equipment, there are two of everything for redundancy... two network paths, two pieces of equipment, etc.
And because they lost admin access to their management network, no one could go in manually and tell the towers to wilt themselves either.
If the towers had just stopped advertising 911 services, the phone would fall back into the network search mode as I described when you have no sim card. It just starts scanning the frequencies until it see's an information block for a network it can talk with the emergency support advertised to and does an emergency attach to the network that the carriers will all accept (An unauthenticated attach for the sole purpose of contacting an emergency center).
So my suspicion is because carriers are so used to we have two of everything, and all emergency calls are marked for priority handling at all layers of the equipment (they get high priority bits on all the network packets and priority CPU scheduling in all the equipment), this particular failure mode where there was a fault somewhere down the line, and they lost control of the towers to tell them to stop advertising 911 services all sort of played together to create the failure mode.
Status: Restored AT&T FINAL, Service Degradation, Global Smart Messaging Suite AT&T Global Smart Messaging Suite
Event description: FINAL, Service Degradation Impacted Services: MMS MT Start time: 02-21-2024 22:00 Eastern, 21:00 Central, 19:00 Pacific End time: 02-22-2024 11:00 Eastern, 10:00 Central, 08:00 Pacific
Downtime: 780 minutes
Dear Customer, We are writing to inform you that Global Smart Messaging Suite is now available. The MMS MT service has been restored and our team is currently monitoring Thank you.
AT&T Business Solutions Kind Regards,
The AT&T SMS Service Administrator