You can contribute to our attempts to find the bad router card here: https://twitter.com/alexstamos/status/1336099461622157312 Almost certainly 12.242.117.22
[deleted]
AT&T Fiber in the SF Bay Area is flipping bits
151–160 of 374 posts
Re: AT&T Fiber in the SF Bay Area is flipping bits
#152You can contribute to our attempts to find the bad router card here: https://twitter.com/alexstamos/status/1336099461622157312 Almost certainly 12.242.117.22
You might want to send an e-mail to the NANOG mailing list: https://mailman.nanog.org/mailman/listinfo/nanog
It’s not impossible that people from AT&T read messages posted there.
Re: AT&T Fiber in the SF Bay Area is flipping bits
#153Earlier quoted context omitted.
The problem is that the idiots the hire to do their "technical" support have zero skill (nor motivation to learn) to assess whether it's a 5% problem or not, and the majority of end-users aren't capable to answer that question either, nor that they are incentivized to answer truthfully. The solution could be a priority support tier where you pay upfront for an hour of a real network engineer's time (decently compensa…
I realize this was written from a position of frustration (which I share) at getting run around by customer support, but I'd reconsider the blanket characterization of tech support staff as "idiots": they're doing a high-throughput job following a playbook they're given with, as you identify, no incentive ---it's probably less about personal motivation than the expectations that are set for how they perform their job…
Good point. The managers would probably give negative feedback to people that took more time on their calls in order to try to help the customer better.
Re: AT&T Fiber in the SF Bay Area is flipping bits
#154Trying to explain this issue to AT&T support is like trying to convince a doctor you're the only person on earth with a particular disease. Even explaining the issue is hard. It's not an outage, my internet isn't out , it's intermittently wrong . The phone support agents aren't prepared for this, and I can't find any way to escalate or speak to a network engineer. I feel like if I spoke to the right engineer, there'd…
When I first moved to Seattle there was a great local ISP called CondoInternet that mainly specialized in high-density downtown buildings. I was once having some packet loss issues and called their support line. I assume the company was really small at the time, because the guy who answered the phone was clearly a network engineer who knew the system inside and out. I read him a couple of traceroutes over the phone a…
XMission in Salt Lake City had support like that, back when DSL unbundling was a thing (2006).
Re: AT&T Fiber in the SF Bay Area is flipping bits
#155I thought that the link layer has built in mechanisms for detecting and correcting errors. Can someone ELI5 what's going on?
Re: AT&T Fiber in the SF Bay Area is flipping bits
#156Earlier quoted context omitted.
The problem is that the idiots the hire to do their "technical" support have zero skill (nor motivation to learn) to assess whether it's a 5% problem or not, and the majority of end-users aren't capable to answer that question either, nor that they are incentivized to answer truthfully. The solution could be a priority support tier where you pay upfront for an hour of a real network engineer's time (decently compensa…
I realize this was written from a position of frustration (which I share) at getting run around by customer support, but I'd reconsider the blanket characterization of tech support staff as "idiots": they're doing a high-throughput job following a playbook they're given with, as you identify, no incentive ---it's probably less about personal motivation than the expectations that are set for how they perform their job…
+1 the real idiots here are att mid-upper management that setup this process and also have zero monitoring for packet loss/bit flips apparently so that they have an outage for weeks now. Support techs have no training nor tooling to debug this issue.
Re: AT&T Fiber in the SF Bay Area is flipping bits
#157If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…
All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…
And conversely its always surprising and disarming when you call a company and actually get through to a knowledgable employee. I was so surprised to hear “thats a firmware bug we know about and there is no update yet” about my router issue that I forgot to be mad at the company for not caring my router is broken.
Re: AT&T Fiber in the SF Bay Area is flipping bits
#158If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…
Why wouldn't the first conclusion be that the example.com service was flipping bits? Load balancing TLS requests to a different front-end pool is very common. Has anyone claimed to see these flipped bits on a domain other than example.com?
”Has anyone claimed to see these flipped bits on a domain other than example.com?“
See e.g. https://gist.github.com/bmastenbrook/14c0e22fc02b95d4a48f82d...
Re: AT&T Fiber in the SF Bay Area is flipping bits
#159Not that I have any specific evidence this is the case, but it wouldnt surprise me to later learn this is an attack on encryption by the NSA.
AT&T in San Francisco is ground 0 for some of their bulk collection.
After The Times disclosed the Bush administration’s warrantless wiretapping program in December 2005, plaintiffs began trying to sue AT&T and the N.S.A. In a 2006 lawsuit, a retired AT&T technician named Mark Klein claimed that three years earlier, he had seen a secret room in a company building in San Francisco where the N.S.A. had installed equipment. [1]
Re: AT&T Fiber in the SF Bay Area is flipping bits
#160Earlier quoted context omitted.
Each TCP packet has a checksum but it does not catch 100% of all possible bit flips
And some routing hardware have been known to ignore this--meaning they'll forward the packet data along and re-calculate the checksum.
Hardware that forwards packets usually forwards IP packets, it doesn't care about TCP checksums and doesn't recalculate them. TCP stack in your OS is the one that does that.
It has to be a special kind of hardware that does much deeper packet inspection (DPI) to recalculate TCP checksums, usually used for spying, throttling, censorship, injecting ads, injecting exploits, etc., but not merely routing/forwarding packets.