AT&T Fiber in the SF Bay Area is flipping bits • Hacker News Board

Re: AT&T Fiber in the SF Bay Area is flipping bits

#121

Post by darig » Tue, Dec 08, 2020, 1:30 AM UTC

cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

Why wouldn't the first conclusion be that the example.com service was flipping bits? Load balancing TLS requests to a different front-end pool is very common.

Has anyone claimed to see these flipped bits on a domain other than example.com?

Re: AT&T Fiber in the SF Bay Area is flipping bits

#122

Post by warent » Tue, Dec 08, 2020, 1:31 AM UTC

dudul wrote: post #120

warent wrote: post #107

Earlier quoted context omitted.

The obnoxious thing to me is the hubris that must be behind this. Either they considered it and decided they would never encounter a system error like this and refused to implement an escalation route, or they never even considered it. Or perhaps even worse than that, maybe they considered it, decided it was possible, but just don't care because of their insane borderline monopoly. I don't understand how internet com…

Wait so you have your main cable plugged into their modem, their router plugged into their modem, and then your own router plugged into their router?

Yes exactly. Their router has a LAN with my router as the only other device, which it's bridged with, and then my router has the true home LAN.

A weird side effect of this is that I'm not using the 192.168.x.x range like usual (because that's what theirs is using), but instead the 10.0.x.x range

Re: AT&T Fiber in the SF Bay Area is flipping bits

#123

Post by beervirus » Tue, Dec 08, 2020, 1:39 AM UTC

cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

Why would a TCP connection allow flipped bits to make it through?

Re: AT&T Fiber in the SF Bay Area is flipping bits

#124

Post by joshcaron » Tue, Dec 08, 2020, 1:39 AM UTC

simcop2387 wrote: post #64

cpach wrote: post #54

Earlier quoted context omitted.

”I would use traceroute to find a common bad point for everyone“ How do you mean? I use traceroute from time to time but I’m not sure how it would apply in a case like this. Feel free to elaborate :)

Take a traceroute from everyone experiencing the problem, look for the common hops among them all. Then compare that list against people not experiancing the problem to find the differences. The finsl set there are good places to start looking. Sweitches and routers along that way could be the cause

Some people on Twitter have started collecting IPs: https://twitter.com/alexstamos/status/1336100299841314817

Re: AT&T Fiber in the SF Bay Area is flipping bits

#125

Post by cpach » Tue, Dec 08, 2020, 1:41 AM UTC

p1necone wrote: post #105

cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…

Obligatory XKCD reference: https://xkcd.com/806/

Re: AT&T Fiber in the SF Bay Area is flipping bits

#126

Post by Nextgrid » Tue, Dec 08, 2020, 1:42 AM UTC

p1necone wrote: post #105

cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…

The problem is that the idiots the hire to do their "technical" support have zero skill (nor motivation to learn) to assess whether it's a 5% problem or not, and the majority of end-users aren't capable to answer that question either, nor that they are incentivized to answer truthfully.

The solution could be a priority support tier where you pay upfront for an hour of a real network engineer's time (decently compensated so that he actually cares about solving the problem) and the charge is refunded only if the problem indeed ends up being on the ISP's side. This should self-regulate as anyone wasting the engineers' time for a simple problem they could resolve themselves would pay for that time.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#127

Post by gamegoblin » Tue, Dec 08, 2020, 1:42 AM UTC

AaronFriel wrote: post #93

Trying to explain this issue to AT&T support is like trying to convince a doctor you're the only person on earth with a particular disease. Even explaining the issue is hard. It's not an outage, my internet isn't out , it's intermittently wrong . The phone support agents aren't prepared for this, and I can't find any way to escalate or speak to a network engineer. I feel like if I spoke to the right engineer, there'd…

When I first moved to Seattle there was a great local ISP called CondoInternet that mainly specialized in high-density downtown buildings.

I was once having some packet loss issues and called their support line. I assume the company was really small at the time, because the guy who answered the phone was clearly a network engineer who knew the system inside and out. I read him a couple of traceroutes over the phone and we resolved the issue within minutes.

I have never experienced such perfect tech support before or after that with any other ISP.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#128

Post by aaroninsf » Tue, Dec 08, 2020, 1:42 AM UTC

renewiltord wrote: post #13

Sonic has their own fibre in San Francisco. Thank god for that. Life's too short for Comcast or AT&T.

Source?

We have Sonic DSL and I see differences using the example.com test; and at installation time I was told I could only get fiber at my location (backside of Bernal) when AT&T put it in and they could lease it.

That was in 2019.

Do they own their own fiber in some areas?

Re: AT&T Fiber in the SF Bay Area is flipping bits

#129

Post by nodesocket » Tue, Dec 08, 2020, 1:43 AM UTC

cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

In the above linked gist, how we we know it's a low level router bit flip instead of some code / programming error in their MITM / dns / JavaScript injection tomfoolery?

Re: AT&T Fiber in the SF Bay Area is flipping bits

#130

Post by boulos » Tue, Dec 08, 2020, 1:43 AM UTC

secalex wrote: post #76

You can contribute to our attempts to find the bad router card here: https://twitter.com/alexstamos/status/1336099461622157312 Almost certainly 12.242.117.22

Indeed.

My mobile.twitter.com traceroute prefers going through that path, as does en.wikipedia.org (both of which have sucked for me) while a Google route (to 172.217.6.78) hops through 12.122.149.186.