Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AT&T Fiber in the SF Bay Area is flipping bits

twitter.com

Post reply on HN
Print view View original

171–180 of 374 posts

Re: AT&T Fiber in the SF Bay Area is flipping bits

#171

Post by toast0 »

cpach wrote: post #158
darig wrote: post #121

Earlier quoted context omitted.

Why wouldn't the first conclusion be that the example.com service was flipping bits? Load balancing TLS requests to a different front-end pool is very common. Has anyone claimed to see these flipped bits on a domain other than example.com?

”Has anyone claimed to see these flipped bits on a domain other than example.com?“ See e.g. https://gist.github.com/bmastenbrook/14c0e22fc02b95d4a48f82d...

This is a pretty useful piece of debugging here. It might be worthwhile to try to get EdgeCast involved, as it could be a broken thing between AT&T and EdgeCast.

Given that (it seems) only AT&T customers are complaining, and (it seems) it only affects servers on EdgeCast.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#172

Post by saagarjha »

GauntletWizard wrote: post #142
Nextgrid wrote: post #126

Earlier quoted context omitted.

The problem is that the idiots the hire to do their "technical" support have zero skill (nor motivation to learn) to assess whether it's a 5% problem or not, and the majority of end-users aren't capable to answer that question either, nor that they are incentivized to answer truthfully. The solution could be a priority support tier where you pay upfront for an hour of a real network engineer's time (decently compensa…

Who decides if it's the ISP's problem? What if it's "Both"? About 20 years ago I got escalated to high-tier Comcast support for an issue that turned out to be a little of A, a little of B: Comcast (Might have been @Home, based on the timing) required that your MAC address be whitelisted as part of their onboarding process. Early home routers had a "MAC Address Clone" feature for precisely this reason. At some point,…

How did you even manage to figure this out? I’ve never gotten anyone on the phone who could possibly help in a situation like this.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#173

Post by ericseppanen »

mjrpes wrote: post #113
cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

It's not just Fiber. It's also V/DSL connections including resold VDSL through a company like Sonic.

I can confirm that on Sonic (resold AT&T) VDSL I am seeing this exact corruption.

Nice find! I had been wondering why I had been seeing odd TLS failure messages recently.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#174

Post by tyu2 »

thamer wrote: post #167
beervirus wrote: post #137

Earlier quoted context omitted.

Checksums. TCP is supposed to provide reliability against data corruption, resend bad packets, etc.

Checksums are often calculated in hardware on the NIC. I have personally seen a network card send packets with corrupted data (corrupted by the network card itself) and valid TCP checksums, computed on the corrupted data.

That's true, but doesn't apply in this case if you assume you received one with wrong checksum, a network card won't recompute it of course.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#175

Post by sneak »

porknubbins wrote: post #157
p1necone wrote: post #105

Earlier quoted context omitted.

All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…

And conversely its always surprising and disarming when you call a company and actually get through to a knowledgable employee. I was so surprised to hear “thats a firmware bug we know about and there is no update yet” about my router issue that I forgot to be mad at the company for not caring my router is broken.

FWIW, this has the hallmarks of an interaction within the context of an abusive interpersonal relationship.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#176

Post by [deleted] »

tlrobinson wrote: post #119
cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

Data point: I’m on Sonic fiber (resold AT&T fiber) and this script has been running without errors for 5+ minutes.

[deleted]

Re: AT&T Fiber in the SF Bay Area is flipping bits

#177

Post by jrmg »

hangonhn wrote: post #36
jmount wrote: post #8

We are AT&T in SF Bay Area, and our service has been flaking out for weeks. Inability to get DNS, machine claims invalid security certificates, download/upload drop below 1 MbS. All we could get is our modem reset.

I'm on Sonic.net fiber and they're a reseller of AT&T and I've experienced this exact scenario.

Same for me on DSLExtreme! I've contacted their support about the issue.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#179

Post by city41 »

ericseppanen wrote: post #173
mjrpes wrote: post #113

Earlier quoted context omitted.

It's not just Fiber. It's also V/DSL connections including resold VDSL through a company like Sonic.

I can confirm that on Sonic (resold AT&T) VDSL I am seeing this exact corruption. Nice find! I had been wondering why I had been seeing odd TLS failure messages recently.

Sonic might be the best avenue to get this fixed. They care about their customers and presumably can talk with AT&T in a much more meaningful way.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#180

Post by sneak »

warent wrote: post #107
AaronFriel wrote: post #93

Trying to explain this issue to AT&T support is like trying to convince a doctor you're the only person on earth with a particular disease. Even explaining the issue is hard. It's not an outage, my internet isn't out , it's intermittently wrong . The phone support agents aren't prepared for this, and I can't find any way to escalate or speak to a network engineer. I feel like if I spoke to the right engineer, there'd…

The obnoxious thing to me is the hubris that must be behind this. Either they considered it and decided they would never encounter a system error like this and refused to implement an escalation route, or they never even considered it. Or perhaps even worse than that, maybe they considered it, decided it was possible, but just don't care because of their insane borderline monopoly. I don't understand how internet com…

Either that or every single CSR takes 50 calls per day, every day, that start with "There's a problem with the AT&T network!!!"
Post reply on HN