Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AT&T Fiber in the SF Bay Area is flipping bits

twitter.com

Post reply on HN
Print view View original

311–320 of 374 posts

Re: AT&T Fiber in the SF Bay Area is flipping bits

#311

Post by jcims »

myself248 wrote: post #307
jcims wrote: post #300

Earlier quoted context omitted.

Amazing story! My step dad worked night shift at AT&T back in the 80’s and ran the 5ESS. He took my brother and I in for a tour one night. Thinking back on it now it was a lean crew for the equipment they were running. Rows and rows and rows of equipment. I don’t remember closed cabinets, mostly open frames moderately populated. I’ll never forget he showed us some magnetic core memory that was still mounted up on a f…

That's super cool! I believe the #5 ESS machine itself is always in closed cabinets, so it's likely that what you're remembering was the toll/transport equipment, or ancillary frames. Gray 23-inch racks as far as the eye can see! Depending on how old that part of the office was, they were likely either 14' or 11'6" tall with rolling ladders in the aisles, or 7' tall and the only place they'd have laddertrack was in f…

You’re definitely on to something. This image from Wikipedia for the #1 ESS fits very well into my fuzzy memory, especially those protruding card chassis:

https://upload.wikimedia.org/wikipedia/commons/7/7e/1A_Frame...

I just remember thinking it looked awkward getting to the equipment under them.

I don’t know if the ‘5E’ as he called it was actually in operation yet, he ended up moving us all out of state to take a job developing and delivering training material for it...I think that’s what finally broke him lol. Hands on kinda dude.

I’ll have to hit him up later today to see if he remembers ‘remreed’ (he will). Thanks for the info!

Re: AT&T Fiber in the SF Bay Area is flipping bits

#312

Post by 14u2c »

m463 wrote: post #223
p1necone wrote: post #105

Earlier quoted context omitted.

All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…

I had a ginormous AT&T router/modem (pace 5268ac) with a set of static ip addresses and a few times, AT&T just stopped routing traffic to it. It had happened before and then magically fixed itself a few days later. One time I had a week of outage with AT&T basically said the problem was on my side. They could ping the modem, and then punted. I had several truck rolls. The techs were really nice guys, but were basical…

That is absurd that AT&T requires the use of a rented gateway for U-verse. I've never had an issue with a another provider refusing to support off the self hardware before, including with a DSL provider, multiple cable companies, and FiOS (Ethernet on ONT).

Re: AT&T Fiber in the SF Bay Area is flipping bits

#313

Post by jcims »

myself248 wrote: post #303
jl6 wrote: post #296

Earlier quoted context omitted.

Great story. > including all the ephemera that would be forgotten when the call disconnects Interesting to know there is information which is not logged. I’m guessing keeping this info, even for a day, would have helped isolate the issue? How did the echo cancellers pass testing?

They passed testing because they had each been individually crossconnected to a test trunk, and test calls force-routed over that trunk. Then to place them in service, the crossconnects were reconfigured to place them at their normal location in the system. The testing was to prove the voice path of each DSP card, and that those cards were wired into the crossconnect properly. All that was true, the failure happened…

Were the cross-connected circuits channelized or individual voice calls (ds0? I can’t remember from my wan days) or something else?

Re: AT&T Fiber in the SF Bay Area is flipping bits

#314

Post by watt »

miahi wrote: post #281
francoisLabonte wrote: post #29

From my professional experience of programming and debugging networking equipment, this could be a switch/router with a buffer with bad memory (stuck bit maybe). The better chips have CRC/Parity/ECC to cover such issues but there are always those magical choke points where the past CRC is tossed and the new one is generated that can leave a gaping hole. The tricky part is how often is this bad memory buffer used... I…

This kind of incident happened to me in a system that was supposed to have high availability. We had failovers for hardware, but it seems that a network device that was supposed to have HA (and was set up to pass the functionality to another device in case of failure) did not have ECC memory. One memory bit got stuck at 0 and the event was not detected at network level, as the data was repacked with a "clean" CRC. Fo…

Wasted an opportunity to implement code that would detect and handle poison-pill messages. Those will happen in any system where queue is involved and there always needs to be an escape hatch to get rid of them. Deleting the queue is too extreme.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#315

Post by myself248 »

jcims wrote: post #311
myself248 wrote: post #307

Earlier quoted context omitted.

That's super cool! I believe the #5 ESS machine itself is always in closed cabinets, so it's likely that what you're remembering was the toll/transport equipment, or ancillary frames. Gray 23-inch racks as far as the eye can see! Depending on how old that part of the office was, they were likely either 14' or 11'6" tall with rolling ladders in the aisles, or 7' tall and the only place they'd have laddertrack was in f…

You’re definitely on to something. This image from Wikipedia for the #1 ESS fits very well into my fuzzy memory, especially those protruding card chassis: https://upload.wikimedia.org/wikipedia/commons/7/7e/1A_Frame... I just remember thinking it looked awkward getting to the equipment under them. I don’t know if the ‘5E’ as he called it was actually in operation yet, he ended up moving us all out of state to take a…

Yup, the #1 used computerized control, but all the switching was still electromechanical, so it sounded like a typewriter factory, especially during busy-hour.

At night, traffic was often low enough that you could hear individual call setups and teardowns, each a cascade of relay actuations rippling from one part of the floor to another. The junctor relays in particular were oddly hefty and made a solid clack, twice per call setup if I recall correctly, once to prove the path by swinging it over it to a test point of some sort, and then again to actually connect it through. On rare occasion, you'd hear a triple-clack as the first path tested bad, an alternate was set up and tested good, and then connected through.

Moments after such a triple-clack, one of the teleprinters would spring to life, spitting out a trouble ticket indicating the failed circuit.

The #5, on the other hand, was completely electronic, time-division switching in the core. The only clicks were the individual line relays responsible for ringing and talk battery, and these were almost silent in comparison. You couldn't learn anything about the health of the machine by just standing in the middle of it and listening, and anyone in possession of a relay contact burnishing tool will tell you in no uncertain terms, that the #5 has no soul.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#316

Post by watt »

bcrl wrote: post #252
francoisLabonte wrote: post #29

From my professional experience of programming and debugging networking equipment, this could be a switch/router with a buffer with bad memory (stuck bit maybe). The better chips have CRC/Parity/ECC to cover such issues but there are always those magical choke points where the past CRC is tossed and the new one is generated that can leave a gaping hole. The tricky part is how often is this bad memory buffer used... I…

Hardware designers basically started making bad decisions on this issue around the time that VLAN tagging was introduced, as well as harware forwarding of IP packets. When VLAN tags are inserted or removed, the CRC of a packet needs to be adjusted to reflect the inserted, removed and/or modified bytes from the VLAN header. Additionally Both the CRC and IP checksum of a packet needs to be adjusted when TTL is decremen…

This is a great explanation of "always those magical choke points where the past CRC is tossed" that parent poster is referencing. Thank you!

Re: AT&T Fiber in the SF Bay Area is flipping bits

#317

Post by arnaudsm »

cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

Imagine if such an issue appeared in a non-techy-area, it could go unnoticed for years.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#318

Post by jcims »

myself248 wrote: post #315
jcims wrote: post #311

Earlier quoted context omitted.

You’re definitely on to something. This image from Wikipedia for the #1 ESS fits very well into my fuzzy memory, especially those protruding card chassis: https://upload.wikimedia.org/wikipedia/commons/7/7e/1A_Frame... I just remember thinking it looked awkward getting to the equipment under them. I don’t know if the ‘5E’ as he called it was actually in operation yet, he ended up moving us all out of state to take a…

Yup, the #1 used computerized control, but all the switching was still electromechanical, so it sounded like a typewriter factory, especially during busy-hour. At night, traffic was often low enough that you could hear individual call setups and teardowns, each a cascade of relay actuations rippling from one part of the floor to another. The junctor relays in particular were oddly hefty and made a solid clack, twice…

YES! He worked third and we were there all night. He pointed those sounds out to us, it was so cool.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#319

Post by enturn »

cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

For me, TLS errors would happen every so often after we had used all our data allowance and our connection was being shaped to a slow speed. Once the speed returned to normal it was fine. I always thought it was the slow speed but maybe there were bugs in the shaping software.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#320

Post by ovi256 »

pakl wrote: post #271
p1necone wrote: post #105

Earlier quoted context omitted.

All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…

That’s why you need to sign up for CSA Pre™. Get preauthorized for instant escalation on customer support calls. All you have to do is answer a form with questions like: Do you know how to plug in a computer? Do you know where the power switches are on your devices? etc. CSA Pre™ is valid for 5 years; you can initiate the renewal process to do 6 months before expiry.

You were joking but I really wished you weren't and this service existed!
Post reply on HN