Live data from Hacker News

AT&T Fiber in the SF Bay Area is flipping bits

twitter.com

191–200 of 374 posts

Re: AT&T Fiber in the SF Bay Area is flipping bits

#191

Sonic has their own fibre in San Francisco. Thank god for that. Life's too short for Comcast or AT&T.

Source? We have Sonic DSL and I see differences using the example.com test; and at installation time I was told I could only get fiber at my location (backside of Bernal) when AT&T put it in and they could lease it. That was in 2019. Do they own their own fiber in some areas?

Sonic has their own fiber in some parts of SF/Santa Rosa and you would know if you were on it, all Sonic DSL products are essentially resold AT&T uverse.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#192

Earlier quoted context omitted.

Take a traceroute from everyone experiencing the problem, look for the common hops among them all. Then compare that list against people not experiancing the problem to find the differences. The finsl set there are good places to start looking. Sweitches and routers along that way could be the cause

Some people on Twitter have started collecting IPs: https://twitter.com/alexstamos/status/1336100299841314817

They need to be capturing src/dest IPs as well as ports for AT&T to have any hope of using that data.

Edit to make the comment more useful: If anyone is curious, look up "ECMP hashing." There are probably tons of parallel paths through AT&T's network, and to narrow down to the hardware causing problems, they will need to identify which specific path was chosen. Hardware switches packets out equally viable pathways by hashing some of the attributes of the packet. Hash output % number of pathways selects which pathway at every hop.

Hardware does this because everyone wants all packets involved in the same "flow" (all packets with the same src/dest IP and port and protocol (TCP)) to deterministically go through the same set of pipes to avoid packet re-ordering. If you randomly sprayed packets, the various buffer depths of routers (or even speed of light and slightly different length fibers along the way) could cause packets to swap ordering. While TCP "copes" with reordering, it doesn't like it and and older implementations slowed way down when it happened.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#193
Some new data.... I ran the same script on two different machines connected via the same AT&T provided modem. Machine A is NATted, B uses a public address. A always gets the error within 50-60 fetches. B doesn't (I ran the test a number of times for a few seconds each). I even ran them both concurrently (repeated the test on A since it keeps failing) but it was always A that got the error.

I am wondering if this is modem related -- may be they silently "upgraded" it or changed some default such as TCP checksum offload? Just guessing, I haven't checked.

Experiment 2: I switched machine A to use a public address (from my static IP address block) and the problem disappeared! I then switched it back to the way it was and still there are no errors. Not sure what to make of this...

Re: AT&T Fiber in the SF Bay Area is flipping bits

#194

Trying to explain this issue to AT&T support is like trying to convince a doctor you're the only person on earth with a particular disease. Even explaining the issue is hard. It's not an outage, my internet isn't out , it's intermittently wrong . The phone support agents aren't prepared for this, and I can't find any way to escalate or speak to a network engineer. I feel like if I spoke to the right engineer, there'd…

When I first moved to Seattle there was a great local ISP called CondoInternet that mainly specialized in high-density downtown buildings. I was once having some packet loss issues and called their support line. I assume the company was really small at the time, because the guy who answered the phone was clearly a network engineer who knew the system inside and out. I read him a couple of traceroutes over the phone a…

And when Wave bought them, Wave put incompetent people in charge. I moved to a Seattle suburb and had "Wave G" (post-Wave CondoInternet).

I had throughput issues, first on my apartment. That got resolved via Reddit. Then the backbone is slow from time to time which persisted. Sometimes I get a Gigabit, other times I only get 10-20 Mbps.

What did Wave say? Let's bring a tech out. I told them "it's a backbone issue" and they didn't understand me. I don't know how I would explain how networking works to a field tech who only knows how to enable a port or run basic diagnostics.

Another time, I tripped Port Security, and told them they can re-enable my port. Wave said "we need to bring a tech out" as if that will magically solve everything.

This is made worse by my apartment's exclusivity deals: only Comcast and Wave, nothing else. Ziply Fiber (bought Frontier) and Atlas Networks were denied.

Bigger companies like Verizon FiOS in NYC had better tech support, I could still get hold of a Level 2 tech for a much smaller issue (and not even on FiOS).

Wave G makes AT&T's forced router (but not the flipping bits) seem decent in comparison. I don't want Comcast but I'll happily take AT&T Fiber over Wave G even if it means trading my pfSense box for a crappy AT&T gateway (worse case scenario I could bypass or root the gateway, or pay $15 for static IPs).

I fortunately moved (for another reason) and have Google's Webpass. Webpass may not give me a full Gigabit, but it's usually 400-800 Mbps all the time and not 10-20 Mbps 95% of the time with an occasional Gigabit. I wish CondoInternet sold to Webpass instead of Wave (I don't want a merger now, but still).

Surprisingly, I may have preferred CenturyLink Fiber if available mainly for GPON over a 60GHz PtP microwave link (less oversubscription!), well unless PPPoE+6rd kills my pfSense. But Webpass works pretty darn well, and I think I'll stay unless CenturyLink suddenly gives me 2 Gbps FTTH or something, or I move.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#195

Earlier quoted context omitted.

Much scarier when you actually see it with your own eyes: $ diff example-* 4c4 Example Domain --- > Example Domain 11c11 background-color: #f0f0f2; 23c23 box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02); 41c41 This domain is for use in illustrative examples in documents. You may usm this --- > This domain is for use in illustrative examples in documents. You may use this

I was under the impression there was enough layers of checksums once you get to the level of UDP/TCP that these kind of single bit flips should be detected and dropped before you get to read them. What's going on here? Is networking gear not calculating checksums like it should?

TCP header does have a checksum that is supposed to check both the header and the payload.

So either the bit corruption is such that is not detected by the checksum

Or ATT is doing something nefarious and touches layer 4 and corrupts the data while doing so

Re: AT&T Fiber in the SF Bay Area is flipping bits

#196
post #193

Some new data.... I ran the same script on two different machines connected via the same AT&T provided modem. Machine A is NATted, B uses a public address. A always gets the error within 50-60 fetches. B doesn't (I ran the test a number of times for a few seconds each). I even ran them both concurrently (repeated the test on A since it keeps failing) but it was always A that got the error. I am wondering if this is m…

How are you bypassing the EAP authentication on your AT&T provided gateway? Are you using https://github.com/jaysoffian/eap_proxy ?

My router is also getting a public IP address, I've set my AT&T gateway to bridge mode, but the AT&T gateway is still a monster-in-the-middle, a tumor I can't excise because it has to be between your router and the ONT or else the fiber turns off.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#197
post #14

When AT&T first did their Fiber rollout in SF, one of the things I remember was they charged you $10 extra if you didn't want them to MITM all your connections to insert JavaScript pointing to their own ads. They rolled this back when folks complained, but I wonder if the relevant infrastructure is still sitting around and mangling packets. I would probably expect this to be some network card or cable or connector is…

What I always wondered about this is, unless AT&T has permission from the copyright holder of each and every Web page so modified to distribute the resulting derivative work, how is this practice not criminal copyright infringement?

Re: AT&T Fiber in the SF Bay Area is flipping bits

#198

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

Data point: I’m on Sonic fiber (resold AT&T fiber) and this script has been running without errors for 5+ minutes.

> resold AT&T fiber

Only in certain areas! Within San Francisco on overhead-cabled blocks (for example) their lines are their own.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#200
post #143

Earlier quoted context omitted.

Much scarier when you actually see it with your own eyes: $ diff example-* 4c4 Example Domain --- > Example Domain 11c11 background-color: #f0f0f2; 23c23 box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02); 41c41 This domain is for use in illustrative examples in documents. You may usm this --- > This domain is for use in illustrative examples in documents. You may use this

The same here. It is always the 0x08 bit. (p-x, d-l, g-o etc.).

I wonder if there’s some 64 line I/O somewhere with a couple traces bridged together, where:

0 and 0 are still 0 and 0,

1 and 1 are still 1 and 1,

but 0 and 1 become 1 and 1 (or 0&0)

and 1 and 0 become 1 and 1 (or 0&0).

I remember seeing this wackiness when I bridged two address lines on an EEPROM with the tiniest amount of solder.

RoHS/tin whiskers strikes again?

Post reply on HN