From Twitter:
> I’m hearing AT&T got their shit together and things are working now. Big thanks to @vikxin and @bmastenbrook for doing the heavy lifting here.
AT&T Fiber in the SF Bay Area is flipping bits
231–240 of 374 posts
Re: AT&T Fiber in the SF Bay Area is flipping bits
#232If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…
I have ATT fiber in Texas and was having issues recently, probably for a few days, where DNS props would just fail(I use Google's DNS), huge pauses in page loads with the occasional just doesn't. Happened over the long weekend IIRC and was sporadic enough that I didn't look into it further. I thought it had largely cleared, but now I'm wondering about some ongoing page load pauses..
Re: AT&T Fiber in the SF Bay Area is flipping bits
#233Earlier quoted context omitted.
All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…
I had a ginormous AT&T router/modem (pace 5268ac) with a set of static ip addresses and a few times, AT&T just stopped routing traffic to it. It had happened before and then magically fixed itself a few days later. One time I had a week of outage with AT&T basically said the problem was on my side. They could ping the modem, and then punted. I had several truck rolls. The techs were really nice guys, but were basical…
The 5268ac pace router is the worst ISP provided router I've ever had, and I've been an Xfinity/Comcast customer, and I've even had a connection in Wyoming. I detailed my experience with it in a review of a third-party router, and found numerous issues along the way [0]. My favorite is that DMZ+ mode, which is what they offer instead of a traditional DMZ mode, just has some weird MTU issue that leads git and other services to break horribly when running behind a third party router. The solution? Don't use DMZ+ mode. Instead, put the router into NAT mode, and then port forward all of the ports to one private IP address. Bonkers. This is sold as an official-looking solution on the AT&T website for a "speed issue." [1]
This is all because AT&T believes that the edge of their network is not the PON ONT/OLT, but rather, the router they issue you. If you want to be on their network, you have to use their router as some part of the chain.
My latest discovery is that in doing this, the router can actually get super hot operating at gigabit speeds for extended periods of time. When this happens, it magically starts dropping packets. Solution? Aim a fan at the router so it has "thermal management."
Total. Garbage. I'd switch to Spectrum if they had decent upload speed, but alas, they don't in my building.
[0]: https://particle17.xyz/posts/amplifi-alien-thoughts/#appendi...
[1]: https://www.att.com/support/article/u-verse-high-speed-inter...
Re: AT&T Fiber in the SF Bay Area is flipping bits
#234Earlier quoted context omitted.
When I first moved to Seattle there was a great local ISP called CondoInternet that mainly specialized in high-density downtown buildings. I was once having some packet loss issues and called their support line. I assume the company was really small at the time, because the guy who answered the phone was clearly a network engineer who knew the system inside and out. I read him a couple of traceroutes over the phone a…
And when Wave bought them, Wave put incompetent people in charge. I moved to a Seattle suburb and had "Wave G" (post-Wave CondoInternet). I had throughput issues, first on my apartment. That got resolved via Reddit. Then the backbone is slow from time to time which persisted. Sometimes I get a Gigabit, other times I only get 10-20 Mbps. What did Wave say? Let's bring a tech out. I told them "it's a backbone issue" an…
If you're interested or can't find it, I'll dig up a link.. but I recently unplugged my AT&T supplied residential gateway after installing a supplicant docker image on my UniFi Dream Machine Pro. It answers the authentication challenges with a reply using a CA from a jailbroken AT&T fiber modem (sourced from a guy on eBay!).
I see you're no longer on AT&T fiber, but if you would use it without their gateway, know that you can!
Re: AT&T Fiber in the SF Bay Area is flipping bits
#235Earlier quoted context omitted.
Wait so you have your main cable plugged into their modem, their router plugged into their modem, and then your own router plugged into their router?
Funnily enough this is how my AT&T fiber is set up as well. They force you to use their router, and you can’t directly connect your router to the ONT. The problem is that they use a device certificate + EAP. There’s work arounds but it’s a pain.
Oh, there's a workaround for that.. I've unplugged my Residential Gateway and now my UniFi dream machine pro is directly connected to the ONT.
You install a CA from a jailbroken modem into a supplicant container that runs on the UDM pro. It confirms to the network that you are using "authorised" equipment for the connection and the packets flow!
Re: AT&T Fiber in the SF Bay Area is flipping bits
#236Earlier quoted context omitted.
It’s amusing seeing Cupertino city council transcripts about this because the people show up claiming 5G gives them cancer and the city council desperately tries to get them to use better excuses so they can approve denying it. Makes a change from the city council members usual practice of denying Vallco permits and claiming Apple employees are hiring prostitutes and molesting high school students. (I did not make th…
Are these transcripts online?
Only informally as far as I know.
https://twitter.com/HousingValley/status/1125959134329708545
https://twitter.com/HousingValley/status/1154781703262498816
Re: AT&T Fiber in the SF Bay Area is flipping bits
#237The script linked in the gist ( https://gist.github.com/bmastenbrook/14c0e22fc02b95d4a48f82d... ) is reporting diffs for me on Sonic fiber as well, which is leased from AT&T in much of the Bay Area. So seems to be a network layer or below issue. Interestingly, the incorrect ASCII chars are always 8 chars higher or lower than their counterpart, indicating that bit 3 is consistently being flipped: flip: f|0b1100110 ori…
I love this. In 1992, I bought a Kenwood stereo receiver, which I still have in my living room to this day, which you could program your radio station call letters into for your presets, and I noticed that after setting one of the presets to "KNBR" it switched to "KNBB". I assumed I made a mistake and corrected it, but a few days later I noticed it happened again. Assuming ASCII encoding, B is 0x42 and R is 0x52, so the difference is the single bit (and even assuming non-ASCII, the delta of 16 is likely still the same), so I experimented and determined that that particular memory location always eventually zeroed the 4th least significant bit, so all characters from P-Z would be shifted down to the corresponding character 16 positions less than it. Nowadays I don't really use the radio presets anymore, but I still like knowing the flaw is there nearly 30 years later.
Re: AT&T Fiber in the SF Bay Area is flipping bits
#238Earlier quoted context omitted.
I had a ginormous AT&T router/modem (pace 5268ac) with a set of static ip addresses and a few times, AT&T just stopped routing traffic to it. It had happened before and then magically fixed itself a few days later. One time I had a week of outage with AT&T basically said the problem was on my side. They could ping the modem, and then punted. I had several truck rolls. The techs were really nice guys, but were basical…
The 5268ac pace router is the worst ISP provided router I've ever had, and I've been an Xfinity/Comcast customer, and I've even had a connection in Wyoming. I detailed my experience with it in a review of a third-party router, and found numerous issues along the way [0]. My favorite is that DMZ+ mode, which is what they offer instead of a traditional DMZ mode, just has some weird MTU issue that leads git and other se…
Apparently you can extract the 802.1x key from the router and then use your own router, and someone even has a script to MITM the connection between the router and ONT.
Re: AT&T Fiber in the SF Bay Area is flipping bits
#239From my professional experience of programming and debugging networking equipment, this could be a switch/router with a buffer with bad memory (stuck bit maybe). The better chips have CRC/Parity/ECC to cover such issues but there are always those magical choke points where the past CRC is tossed and the new one is generated that can leave a gaping hole. The tricky part is how often is this bad memory buffer used... I…
Why would a switch/router recalculate and rewrite the TCP checksum?
because they're changing the payload.
Re: AT&T Fiber in the SF Bay Area is flipping bits
#240Earlier quoted context omitted.
The checksum should be checked by your computer. So somehow the packet is being repackaged with the correct checksum, but for the wrong data. In other words, when your computer checks the checksum, it matches. Another possibility is that somehow only errors that result in the same checksum are being generated.
Or their computer isn't checking the checksum. As is apparently the case on mac os (as reported elsewhere in the thread). Another quality product by Apple ;P
Except it was trivial to reproduce with the script on non-Apple devices, and people in one of the many Twitter threads surrounding this showed that on their Mac there was MANY tcp retransmits due to invalid checksums, and the bit-flipped packet did have the correct checksum.