Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cellular outage in U.S. hits AT&T, T-Mobile and Verizon users

cnbc.com

Post reply on HN
Print view View original

431–440 of 486 posts

Re: Cellular outage in U.S. hits AT&T, T-Mobile and Verizon users

#431

Post by bronco21016 »

topkai22 wrote: post #424
jrochkind1 wrote: post #418

Earlier quoted context omitted.

I've been thinking about how pretty much no USA infrastructure of today is as reliable as of the last half of the 20th century. Just my imagination, or true? And what does it mean?

Airliners crash far less, road fatalities kept going down till 2010, so taking the broad view of were infrastructure is I’d say at least those got better. Anecdotally, I remember more electricity interruptions and plumbing issues when I was a kid, but that could be location dependent and I couldn’t quickly find good numbers going back that far. Edit: While the phone network didn’t necessarily go down, I frequently go…

I wish we had metrics for utility companies. In my midwestern experience, things have gotten worse. I don’t remember any outages as a kid in the 90s that were over 24 hours aside from the major blackout in the early 2000s. As an adult I’ve experienced several outages greater than 24 hours in both summer and winter months. It’d be nice to be able to measure this.

Re: Cellular outage in U.S. hits AT&T, T-Mobile and Verizon users

#432

Post by dymk »

lor_louis wrote: post #420
jrochkind1 wrote: post #418

Earlier quoted context omitted.

I've been thinking about how pretty much no USA infrastructure of today is as reliable as of the last half of the 20th century. Just my imagination, or true? And what does it mean?

Redundancy was needed because individual nodes/machines were more prone to failure. As machines got more and more reliable, having highly redundant infrastructure was seen as an extra cost.

Do you work at a telecom or are you just guessing?

Re: Cellular outage in U.S. hits AT&T, T-Mobile and Verizon users

#433

Post by danlugo92 »

Animats wrote: post #331

This is a serious architectural flaw. In the entire history of electromechanical switching in the Bell System, no central office was ever out of service for more than 30 minutes for any reason other than a natural disaster, or, on one single occasion, a major fire in NYC. The AT&T Long Lines system in the 1960s and 1970s had ten regional centers, all independent and heavily interconnected. There was a control center,…

True words :checkmark

Re: Cellular outage in U.S. hits AT&T, T-Mobile and Verizon users

#434

Post by pests »

bobthepanda wrote: post #411
steelframe wrote: post #402

Earlier quoted context omitted.

> Traffic worsened because drivers weren't able to optimize their routes This might explain a huge random traffic jam I hit in the middle of my town this morning. I had no idea any kind of an outage was happening because I've intentionally scaled back my dependence on my phone. I always used to automatically pull up Google Maps to navigate no matter how short the trip. At some point I realized I was losing my ability…

there is also a big of fear, that if you go to an alternate route it may also be congested, and it will all have been for nothing. radio is pretty good with traffic news, but how many people would even think of local radio?

I think many many people would check local radio, not everyone lives on their phone or in a tech bubble.

Re: Cellular outage in U.S. hits AT&T, T-Mobile and Verizon users

#435

Post by Aloisius »

bronco21016 wrote: post #431
topkai22 wrote: post #424

Earlier quoted context omitted.

Airliners crash far less, road fatalities kept going down till 2010, so taking the broad view of were infrastructure is I’d say at least those got better. Anecdotally, I remember more electricity interruptions and plumbing issues when I was a kid, but that could be location dependent and I couldn’t quickly find good numbers going back that far. Edit: While the phone network didn’t necessarily go down, I frequently go…

I wish we had metrics for utility companies. In my midwestern experience, things have gotten worse. I don’t remember any outages as a kid in the 90s that were over 24 hours aside from the major blackout in the early 2000s. As an adult I’ve experienced several outages greater than 24 hours in both summer and winter months. It’d be nice to be able to measure this.

I would caution comparing today against one's childhood memories.

Children have few responsibilities and are shielded by their caretakers. They simply do not notice much of the things that happen.

Re: Cellular outage in U.S. hits AT&T, T-Mobile and Verizon users

#437

Post by Animats »

lor_louis wrote: post #420
jrochkind1 wrote: post #418

Earlier quoted context omitted.

I've been thinking about how pretty much no USA infrastructure of today is as reliable as of the last half of the 20th century. Just my imagination, or true? And what does it mean?

Redundancy was needed because individual nodes/machines were more prone to failure. As machines got more and more reliable, having highly redundant infrastructure was seen as an extra cost.

Yes. Electromechanical switching systems were substantially more reliable than their components. How this was done should be understood by anybody designing high-reliability systems today.

"A History of Science and Engineering in the Bell System - Switching Technology 1925-1975" is a readable reference. The Internet Archive has it.[1]

More hardcore: "No. 5 Crossbar"[2]

The Connections Museum in Seattle still has a #5 Crossbar working.[3] Long distance used toll switches, "#4 Crossbar", and there were 202 of them.

#4 and #5 Crossbar machines are collections of stateless microservices, implemented from electromechanical components. The terminology is used in the old books is completely different, but that's what they are. Each service always has at least two servers. The parts that do have state are distributed. The crossbar switches that make actual connections have state, but are dumb - they are told what to do by "markers", which are stateless but can read the state of the crossbars and of other components. Failure of a single crossbar unit can take down less than a hundred lines at most. Other than the crossbars to external lines, everything had alternate routes. Everything has fault detection, with lights and alarm bells.

Error rates were fairly high. In the previous "step by step" system, a good central office misdirected about 1% of calls. With bad maintenance (and those things were high maintenance) that could get much worse. Crossbar was better, maybe 0.1% misdirected calls.

Routing tables in crossbar were mostly static ROMs of one kind or another. Routing consisted of trying a predetermined set of routes, in order. Clunky, but reliable.

Modern systems need a backdown to that mode.

[1] https://archive.org/details/historyofenginee0000unse_q0d8

[2] https://archive.org/details/bellsystem-no-5-crossbar-blr

[3] http://www.telcomhistory.org/connections-museum-seattle-exhi...

Re: Cellular outage in U.S. hits AT&T, T-Mobile and Verizon users

#438

Post by onthecanposting »

jrochkind1 wrote: post #418
Animats wrote: post #331

This is a serious architectural flaw. In the entire history of electromechanical switching in the Bell System, no central office was ever out of service for more than 30 minutes for any reason other than a natural disaster, or, on one single occasion, a major fire in NYC. The AT&T Long Lines system in the 1960s and 1970s had ten regional centers, all independent and heavily interconnected. There was a control center,…

I've been thinking about how pretty much no USA infrastructure of today is as reliable as of the last half of the 20th century. Just my imagination, or true? And what does it mean?

Highway construction standards are much higher than they were 30-50 years ago, but it's a mixed bag. Administrative costs are significantly higher. Survey has dramatically improved with GPS. Highway engineering has not improved since about the 90s. Automated machine guidance has significantly improved the potential accuracy of grading operations in th last decade.

Re: Cellular outage in U.S. hits AT&T, T-Mobile and Verizon users

#439

Post by anitil »

kevin_nisbet wrote: post #200

To everyone trying to speculate on the root cause, I haven't seen enough information in any of the comments to really draw any conclusions. Having worked on several nationwide cellular issues in Canada when I worked in telecom, we saw nationwide impacts based on any number of causes. - A new route injected in the network caused the routing engines on a type of cellular specific equipment to crash nationwide. This too…

> All the CRTC filings I reviewed had all the useful information redacted

Is this common in the industry?

Re: Cellular outage in U.S. hits AT&T, T-Mobile and Verizon users

#440

Post by spacebacon »

Animats wrote: post #331

This is a serious architectural flaw. In the entire history of electromechanical switching in the Bell System, no central office was ever out of service for more than 30 minutes for any reason other than a natural disaster, or, on one single occasion, a major fire in NYC. The AT&T Long Lines system in the 1960s and 1970s had ten regional centers, all independent and heavily interconnected. There was a control center,…

I’m surprised no one noticed the 502 when attempting to enable WiFi calling. Azure was the source of the 502. Cloud architecture problem.
Post reply on HN