AT&T Fiber in the SF Bay Area is flipping bits • Hacker News Board

Re: AT&T Fiber in the SF Bay Area is flipping bits

#351

Post by miahi » Tue, Dec 08, 2020, 5:38 PM UTC

watt wrote: post #314

miahi wrote: post #281

Earlier quoted context omitted.

This kind of incident happened to me in a system that was supposed to have high availability. We had failovers for hardware, but it seems that a network device that was supposed to have HA (and was set up to pass the functionality to another device in case of failure) did not have ECC memory. One memory bit got stuck at 0 and the event was not detected at network level, as the data was repacked with a "clean" CRC. Fo…

Wasted an opportunity to implement code that would detect and handle poison-pill messages. Those will happen in any system where queue is involved and there always needs to be an escape hatch to get rid of them. Deleting the queue is too extreme.

Deleting the queues is an operational decision that I made to be able to put the system back online after the network device was replaced (the important part was the uptime/SLA). From a quick analysis of the logs the percentage of bad messages was ~90% (there was a ~50% chance that the original "touched" bit was 0 so no change was done, but the messages had multiple "touched" bits at fixed intervals).

There was an escape hatch, but the conditions to hit it were a bit complex. Implementing new message filtering of this kind at 2AM while the system was down was not feasible.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#352

Post by proverbialbunny » Tue, Dec 08, 2020, 5:48 PM UTC

city41 wrote: post #188

proverbialbunny wrote: post #65

Earlier quoted context omitted.

It's because AT&T does not follow net neutrality. It has fast lanes and slow lanes, which cause a lot of problems for work VPNs even when down the street. This TLS handshake bit flip is just yet another issue AT&T has, including the 1.1.1.1 DNS issues, and everything else. I'm on Sonic.net fiber over AT&T too. To get the equivalent fiber from Comcast out here it's $270 a month. However, if you're lucky enough Sonic.n…

Huh, I had Sonic fiber over AT&T and loved it. In May I moved out of the bay area and now have Comcast cable. I'd do pretty much anything to get Sonic's level of quality out here.

That's because you're comparing fiber to cable. Get Comcast fiber if you want it again. I don't know if $270 a month is a statewide rate or not.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#353

Post by maerF0x0 » Tue, Dec 08, 2020, 6:10 PM UTC

Bluecobra wrote: post #350

maerF0x0 wrote: post #159

Not that I have any specific evidence this is the case, but it wouldnt surprise me to later learn this is an attack on encryption by the NSA. AT&T in San Francisco is ground 0 for some of their bulk collection. After The Times disclosed the Bush administration’s warrantless wiretapping program in December 2005, plaintiffs began trying to sue AT&T and the N.S.A. In a 2006 lawsuit, a retired AT&T technician named Mark…

I would think that if that room did/still exists they would most likely use passive optical taps, not interfere with transit traffic. Optical taps simply mirror a fiber optic signal one way to a tap aggregation switch/packet capture devices. They would also be mostly undetectable, you will just see a reduced light level.

I'm pointing out that flipping bits could be a form of cryptographic attack (potentially against an implementation bug, not necessarily against the math/algo) .

For example flipping a bit possibly could cause a checksum failure and retransmit.

I'm not saying something _is_ happening, mostly just musing on the possibilities.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#354

Post by bdd » Tue, Dec 08, 2020, 6:43 PM UTC

Dane Jasper, CEO of Sonic, which resells AT&T's fiber service in some areas where they don't have their own fiber, responded to a tweet asking if he could help with pushing this to a remedy with AT&T:

> Yep, we are engaged. And they are clueless. Huge challenge to find someone who really has ability to troubleshoot."

https://twitter.com/dane/status/1336111107430207488

Re: AT&T Fiber in the SF Bay Area is flipping bits

#355

Post by noncoml » Tue, Dec 08, 2020, 6:54 PM UTC

ericbarrett wrote: post #276

cranekam wrote: post #261

Earlier quoted context omitted.

TCP checksums are notoriously weak: https://www.evanjones.ca/tcp-checksums.html With enough packets passing the dodgy RAM a noticeable number will manage to get mangled in such a way that the checksum is still correct.

Checksums are also often recomputed on transit if the packet is intercepted, e.g. to limit TTL, unflag odd/unused TCP features, that kind of ISP-ish preening. So if it was a software error or even a hardware problem in the right (wrong) spot it’s possible to get this kind of corruption without retransmits.

I think you are confusing the IP header with TCP header.

Routers don’t touch the TCP header at all

Re: AT&T Fiber in the SF Bay Area is flipping bits

#356

Post by warent » Tue, Dec 08, 2020, 7:11 PM UTC

_rs wrote: post #244

warent wrote: post #122

Earlier quoted context omitted.

Yes exactly. Their router has a LAN with my router as the only other device, which it's bridged with, and then my router has the true home LAN. A weird side effect of this is that I'm not using the 192.168.x.x range like usual (because that's what theirs is using), but instead the 10.0.x.x range

So are you bridged then or is it really a double nat?

this is really where my limited knowledge of networking shows. I'm not entirely sure but I want to say both or double nat. There's two networks, but my router thinks theirs is a modem and is connected via the "Internet" port, not just a normal device port

Re: AT&T Fiber in the SF Bay Area is flipping bits

#357

Post by 35fbe7d3d5b9 » Tue, Dec 08, 2020, 7:32 PM UTC

yrral wrote: post #70

For this type of issue, I would recommend trying the nanog mailing list. You often see network admins ask for someone "with clue" at a different company when they get the runaround by tier 1/2 tech support. https://archive.nanog.org/list/join https://seclists.org/nanog/2019/Aug/103

This is absolutely the right way to "informally" escalate things to people that know what they're doing at big ISPs.

I'd recommend most newbies to the list show their work and post what's broken and how you know it isn't your fault. The investigative work on that Twitter thread is top notch and would do it in a second.

NANOG and outages@ are the two mailing lists that I've been subscribed to forever and are indispensable if you do operations.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#358

Post by todd_m » Tue, Dec 08, 2020, 8:22 PM UTC

Similar problem, on the peninsula, running ATT Fiber via Sonic. There is a known issue with 2 part numbers of ONT that are incompatible with ATT main station, resulting in packet loss, accrued errors, etc. An ATT service tech can come out to replace this in 3 minutes. Have had zero problems since replacement, after getting knocked off network many times per hour.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#359

Post by bytesandbots » Tue, Dec 08, 2020, 8:51 PM UTC

cranekam wrote: post #261

willis936 wrote: post #228

Earlier quoted context omitted.

I don’t have numbers off hand, but from a feeling I have from memory, I would think it is extremely unlikely that TCP checksums are consistently failing to trigger retransmission. Someone must be altering packets along the way.

TCP checksums are notoriously weak: https://www.evanjones.ca/tcp-checksums.html With enough packets passing the dodgy RAM a noticeable number will manage to get mangled in such a way that the checksum is still correct.

In that case, won't there be significant packet loss causing throughput to be very slow? I don't know if this is possible without something messing with TCP headers.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#360

Post by kayfox » Tue, Dec 08, 2020, 9:25 PM UTC

myself248 wrote: post #307

jcims wrote: post #300

Earlier quoted context omitted.

Amazing story! My step dad worked night shift at AT&T back in the 80’s and ran the 5ESS. He took my brother and I in for a tour one night. Thinking back on it now it was a lean crew for the equipment they were running. Rows and rows and rows of equipment. I don’t remember closed cabinets, mostly open frames moderately populated. I’ll never forget he showed us some magnetic core memory that was still mounted up on a f…

That's super cool! I believe the #5 ESS machine itself is always in closed cabinets, so it's likely that what you're remembering was the toll/transport equipment, or ancillary frames. Gray 23-inch racks as far as the eye can see! Depending on how old that part of the office was, they were likely either 14' or 11'6" tall with rolling ladders in the aisles, or 7' tall and the only place they'd have laddertrack was in f…

Theres a telco museum in Seattle called the Connections Museum, it has working panel, #1 crossbar and #5 crossbar switches and a #3ESS they are working on getting running again.

https://www.telcomhistory.org/connections-museum-seattle/