AT&T Fiber in the SF Bay Area is flipping bits • Hacker News Board

Re: AT&T Fiber in the SF Bay Area is flipping bits

#161

Post by bakul » Tue, Dec 08, 2020, 2:31 AM UTC

darig wrote: post #121

cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

Why wouldn't the first conclusion be that the example.com service was flipping bits? Load balancing TLS requests to a different front-end pool is very common. Has anyone claimed to see these flipped bits on a domain other than example.com?

I see no errors running the same script from another host that doesn't go through at&t.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#162

Post by bfirsh » Tue, Dec 08, 2020, 2:33 AM UTC

If you've got AT&T fiber via Sonic, give them a call and ask them to escalate the issue. The people on the phone know about the issue, take you seriously, and the more people complaining the more pressure they'll put on AT&T, I expect.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#163

Post by sagarm » Tue, Dec 08, 2020, 2:36 AM UTC

Nextgrid wrote: post #126

p1necone wrote: post #105

Earlier quoted context omitted.

All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…

The problem is that the idiots the hire to do their "technical" support have zero skill (nor motivation to learn) to assess whether it's a 5% problem or not, and the majority of end-users aren't capable to answer that question either, nor that they are incentivized to answer truthfully. The solution could be a priority support tier where you pay upfront for an hour of a real network engineer's time (decently compensa…

To be fair, the vast majority of technical support calls likely require only customer service rather than technical skills. Anyone with technical chops probably wouldn't last long in that environment.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#164

Post by ribosometronome » Tue, Dec 08, 2020, 2:37 AM UTC

aaroninsf wrote: post #128

renewiltord wrote: post #13

Sonic has their own fibre in San Francisco. Thank god for that. Life's too short for Comcast or AT&T.

Source? We have Sonic DSL and I see differences using the example.com test; and at installation time I was told I could only get fiber at my location (backside of Bernal) when AT&T put it in and they could lease it. That was in 2019. Do they own their own fiber in some areas?

At the very least, Sonic did their own roll out back in 2015 in the Sunset. Saw the trucks and was a very happy customer. https://twitter.com/sonic/status/748371538265649153?lang=en

Re: AT&T Fiber in the SF Bay Area is flipping bits

#165

Post by gwittel » Tue, Dec 08, 2020, 2:39 AM UTC

beervirus wrote: post #123

cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

Why would a TCP connection allow flipped bits to make it through?

Could be crappy/buggy middle boxes. Especially if they’re inspecting packets or messing with SSL.

My ATT modem (Arris bgw-210-700) may have gotten a firmware upgrade recently as I found some settings I shut off got re-enabled. I use it for VDSL but the same model family is used for ATT fiber.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#166

Post by fomine3 » Tue, Dec 08, 2020, 2:40 AM UTC

mschuster91 wrote: post #49

cameldrv wrote: post #40

Earlier quoted context omitted.

The parts of SV that haven't buried their power lines (not too pretty to begin with) have gotten significantly uglier since all of the new cable/fiber/DSL infrastructure has gone in. There are in many cases multiple of these things on loads of poles: https://www.dslreports.com/forum/r31593060-Equip-What-is-thi...

Here in Germany we bury everything . I don't even want to imagine how long it takes to rebuild all of that when some drunkard crashes his car into one of these poles or a lightning strike hits.

Here in Japan, there's full of pole except a few tourist spot, recovery after car accident, earthquake or heavy rain is done immediately.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#167

Post by thamer » Tue, Dec 08, 2020, 2:41 AM UTC

beervirus wrote: post #137

cpach wrote: post #132

Earlier quoted context omitted.

How do you mean? How would it know if a bit in the HTTP payload got flipped?

Checksums. TCP is supposed to provide reliability against data corruption, resend bad packets, etc.

Checksums are often calculated in hardware on the NIC. I have personally seen a network card send packets with corrupted data (corrupted by the network card itself) and valid TCP checksums, computed on the corrupted data.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#168

Post by fomine3 » Tue, Dec 08, 2020, 2:45 AM UTC

jedberg wrote: post #82

km3r wrote: post #81

Why doesnt TCP reject the flip bits when the checksum fails?

Apparently MacOS doesn't check the checksums.

This is surprising.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#169

Post by rob-olmos » Tue, Dec 08, 2020, 2:47 AM UTC

beervirus wrote: post #137

cpach wrote: post #132

Earlier quoted context omitted.

How do you mean? How would it know if a bit in the HTTP payload got flipped?

Checksums. TCP is supposed to provide reliability against data corruption, resend bad packets, etc.

Depending on TCP checksums and experiencing single-bit flips took down AWS S3 back in the day: https://status.aws.amazon.com/s3-20080720.html

Re: AT&T Fiber in the SF Bay Area is flipping bits

#170

Post by MarkSweep » Tue, Dec 08, 2020, 2:50 AM UTC

francoisLabonte wrote: post #29

From my professional experience of programming and debugging networking equipment, this could be a switch/router with a buffer with bad memory (stuck bit maybe). The better chips have CRC/Parity/ECC to cover such issues but there are always those magical choke points where the past CRC is tossed and the new one is generated that can leave a gaping hole. The tricky part is how often is this bad memory buffer used... I…

To your point, my traceroutes on this problem often have NTT in them. It’s mostly Japanese websites, but also Wikipedia.