Live data from Hacker News

AT&T Fiber in the SF Bay Area is flipping bits

twitter.com

111–120 of 374 posts

Re: AT&T Fiber in the SF Bay Area is flipping bits

#112

From my professional experience of programming and debugging networking equipment, this could be a switch/router with a buffer with bad memory (stuck bit maybe). The better chips have CRC/Parity/ECC to cover such issues but there are always those magical choke points where the past CRC is tossed and the new one is generated that can leave a gaping hole. The tricky part is how often is this bad memory buffer used... I…

nah I'd rather attribute this to malice

Re: AT&T Fiber in the SF Bay Area is flipping bits

#113

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

It's not just Fiber. It's also V/DSL connections including resold VDSL through a company like Sonic.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#114

It’s certainly a relief to have an apparent explanation for the weirdness I’ve been experiencing! I thought it was my home router or WiFi or something, but then I experienced the same problems at my partner’s house, so I thought maybe it’s just a problem with some websites I frequent or some common internet infrastructure. But we both have AT&T internet, so this must be it! For me, the problem has mostly manifested a…

I've also been having this issue with ATT Fiber, but I'm out of Atlanta.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#115

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

Oddly, I can't even resolve example.com (using AT&T DNS).

Sounds like the fixed the “glitch” of excessive traffic against example.com. :)

Re: AT&T Fiber in the SF Bay Area is flipping bits

#116
post #5

Oh man, I thought it was just my crappy old router causing the problems! I've just been too lazy to call them to replace it. I'm going to tweet at them right now. Edit: Reading the tweet thread, classic interaction: ATT says "please click this to check your connection", guy replies back and says "I think we know more about networks that you do, please get a network admin in here".

It's happening to me too! I bought a new router a couple days ago because I couldn't figure out what else could be causing random, sporadic slowdowns.

I can't even download the page 3 times in a row w/o corruption

   4c4
   Example Domain
   ---
   >     Example Domain

Re: AT&T Fiber in the SF Bay Area is flipping bits

#117

Trying to explain this issue to AT&T support is like trying to convince a doctor you're the only person on earth with a particular disease. Even explaining the issue is hard. It's not an outage, my internet isn't out , it's intermittently wrong . The phone support agents aren't prepared for this, and I can't find any way to escalate or speak to a network engineer. I feel like if I spoke to the right engineer, there'd…

[deleted]

Re: AT&T Fiber in the SF Bay Area is flipping bits

#118

Trying to explain this issue to AT&T support is like trying to convince a doctor you're the only person on earth with a particular disease. Even explaining the issue is hard. It's not an outage, my internet isn't out , it's intermittently wrong . The phone support agents aren't prepared for this, and I can't find any way to escalate or speak to a network engineer. I feel like if I spoke to the right engineer, there'd…

From a business standpoint, it's hard to justify paying support to be technical enough to diagnose an issue such as this. Let's be honest, even senior network engineers would have a hard time debugging and diagnosing this. AT&T doesn't want to pay support staff six figure salaries and I assume most senior network engineers don't want to be support agents (customer facing).

AT&T (though applies to lost of companies) probably need a "unicorn" role of a very technical person that is paid as such, but able to interface with customers on specific highly technical issues.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#119

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

Data point: I’m on Sonic fiber (resold AT&T fiber) and this script has been running without errors for 5+ minutes.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#120
post #107

Trying to explain this issue to AT&T support is like trying to convince a doctor you're the only person on earth with a particular disease. Even explaining the issue is hard. It's not an outage, my internet isn't out , it's intermittently wrong . The phone support agents aren't prepared for this, and I can't find any way to escalate or speak to a network engineer. I feel like if I spoke to the right engineer, there'd…

The obnoxious thing to me is the hubris that must be behind this. Either they considered it and decided they would never encounter a system error like this and refused to implement an escalation route, or they never even considered it. Or perhaps even worse than that, maybe they considered it, decided it was possible, but just don't care because of their insane borderline monopoly. I don't understand how internet com…

Wait so you have your main cable plugged into their modem, their router plugged into their modem, and then your own router plugged into their router?
Post reply on HN