Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AT&T Fiber in the SF Bay Area is flipping bits

twitter.com

Post reply on HN
Print view View original

221–230 of 374 posts

Re: AT&T Fiber in the SF Bay Area is flipping bits

#221

Post by anon946 »

p1necone wrote: post #185
noncoml wrote: post #133

Earlier quoted context omitted.

Much scarier when you actually see it with your own eyes: $ diff example-* 4c4 Example Domain --- > Example Domain 11c11 background-color: #f0f0f2; 23c23 box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02); 41c41 This domain is for use in illustrative examples in documents. You may usm this --- > This domain is for use in illustrative examples in documents. You may use this

I was under the impression there was enough layers of checksums once you get to the level of UDP/TCP that these kind of single bit flips should be detected and dropped before you get to read them. What's going on here? Is networking gear not calculating checksums like it should?

The checksum should be checked by your computer. So somehow the packet is being repackaged with the correct checksum, but for the wrong data. In other words, when your computer checks the checksum, it matches. Another possibility is that somehow only errors that result in the same checksum are being generated.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#222

Post by toast0 »

p1necone wrote: post #105
cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…

> If there's anyone out there designing tech support procedures, you should add an "is this a 5% problem?" question to whatever checklist you give to support staff.

When I was the engineer customer service escalated to, I was damn sure to thank them every time they escalated something. Even the one guy who escalated all the things I'd roll my eyes about in private. At least he was making sure the escalation path worked.

Someone who has taken the time to report an issue is probably one of hundreds or thousands who had an issue and didn't think it could be fixed and shrugged it off. We certainly can't fix everything, but weird network shit like this can be fixed, and it's worth escalating, because when you get it fixed, you can also figure out (hopefully) how to monitor for it, so it doesn't happen again.

OTOH, I didn't work for the phone company. We don't care, we don't have to, we're the phone company. https://vimeo.com/355556831 (sorry about the quality, I guess internet video was pretty lowdef in the 70s :P)

Re: AT&T Fiber in the SF Bay Area is flipping bits

#223

Post by m463 »

p1necone wrote: post #105
cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…

I had a ginormous AT&T router/modem (pace 5268ac) with a set of static ip addresses and a few times, AT&T just stopped routing traffic to it.

It had happened before and then magically fixed itself a few days later.

One time I had a week of outage with AT&T basically said the problem was on my side. They could ping the modem, and then punted. I had several truck rolls. The techs were really nice guys, but were basically cabling guys, better for finding a bad cable than debugging a packet loss. The problem for me was that my ipv4 static ip addresses would not receive traffic.

I was at wit's end after a week and I debugged the thing myself. By looking at EVERY bit of data on the router, I found mention of the blocked packets in the firewall log. I would clear all the logs, and found even with the firewall DISABLED, the firewall log would see and block incoming packets I was sending using my neighbor's comcast connection.

I called AT&T, but this time mentioning "firewall is completely off, but packets are blocked by the router and showing up in the log" was concrete enough for them to look up a (known) solution.

The fix was to disable the firewall, but to enable stealth mode. wtf?

To be clear, this was a firmware bug, and caused dozens of calls to AT&T, lots of heartache and finger pointing always in my direction.

I should also mention at the start of this fiasco, I checked the system log and noticed they pushed a firmware update to the modem at the time the problem started. Strangely after one call to the agent, that specific line disappeared out of the log file, but other log entries remained. hmmm.

since then, they basically screw up my modem every month or two - they push new firmware and new "features" appear (like the one that sniffs and categorizes application traffic like "youtube" and "github"). It also helpfully turns wifi BACK ON when I had disabled it. I immediately go turn if back off and then they immediately send me a big warning email that my DSL settings have been changed.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#224

Post by toast0 »

anon946 wrote: post #221
p1necone wrote: post #185

Earlier quoted context omitted.

I was under the impression there was enough layers of checksums once you get to the level of UDP/TCP that these kind of single bit flips should be detected and dropped before you get to read them. What's going on here? Is networking gear not calculating checksums like it should?

The checksum should be checked by your computer. So somehow the packet is being repackaged with the correct checksum, but for the wrong data. In other words, when your computer checks the checksum, it matches. Another possibility is that somehow only errors that result in the same checksum are being generated.

Or their computer isn't checking the checksum. As is apparently the case on mac os (as reported elsewhere in the thread).

Another quality product by Apple ;P

Re: AT&T Fiber in the SF Bay Area is flipping bits

#225

Post by toast0 »

AaronFriel wrote: post #196
bakul wrote: post #193

Some new data.... I ran the same script on two different machines connected via the same AT&T provided modem. Machine A is NATted, B uses a public address. A always gets the error within 50-60 fetches. B doesn't (I ran the test a number of times for a few seconds each). I even ran them both concurrently (repeated the test on A since it keeps failing) but it was always A that got the error. I am wondering if this is m…

How are you bypassing the EAP authentication on your AT&T provided gateway? Are you using https://github.com/jaysoffian/eap_proxy ? My router is also getting a public IP address, I've set my AT&T gateway to bridge mode, but the AT&T gateway is still a monster-in-the-middle, a tumor I can't excise because it has to be between your router and the ONT or else the fiber turns off.

When I was in the service area, I had written a EAP proxy and did it that way (I've still got the code around somewhere, ran on FreeBSD, email in my profile if you want it), but I've heard the easier way is to put an unmanaged switch between the ONT and the residential gateway, let the gateway do the 802.11x, then unplug it from the switch and use hardware of your choice. As long as your switch isn't too smart, it won't know to send an 802.11x signoff, and the ONT will leave the network port working. Yay, security theater.

You may need to spoof mac for DHCP, but I don't know if that's strictly required.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#226

Post by peteretep »

p1necone wrote: post #105
cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…

> If there's anyone out there designing tech support procedures, you should add an "is this a 5% problem?" question to whatever checklist you give to support staff.

... only if you want people to actually be able to escalate. Suspect AT&T are going to lose a lot less money over this issue than hiring an extra high quality support person would cost

Re: AT&T Fiber in the SF Bay Area is flipping bits

#227

Post by snotrockets »

tuzakey wrote: post #191
aaroninsf wrote: post #128

Earlier quoted context omitted.

Source? We have Sonic DSL and I see differences using the example.com test; and at installation time I was told I could only get fiber at my location (backside of Bernal) when AT&T put it in and they could lease it. That was in 2019. Do they own their own fiber in some areas?

Sonic has their own fiber in some parts of SF/Santa Rosa and you would know if you were on it, all Sonic DSL products are essentially resold AT&T uverse.

In some parts of the East Bay as well, but I can't find exact maps. If they sell you "Fusion IP Broadband", it's rebranded AT&T. If it's "Fusion Fiber", it's Sonic's own.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#228

Post by willis936 »

noncoml wrote: post #195
p1necone wrote: post #185

Earlier quoted context omitted.

I was under the impression there was enough layers of checksums once you get to the level of UDP/TCP that these kind of single bit flips should be detected and dropped before you get to read them. What's going on here? Is networking gear not calculating checksums like it should?

TCP header does have a checksum that is supposed to check both the header and the payload. So either the bit corruption is such that is not detected by the checksum Or ATT is doing something nefarious and touches layer 4 and corrupts the data while doing so

I don’t have numbers off hand, but from a feeling I have from memory, I would think it is extremely unlikely that TCP checksums are consistently failing to trigger retransmission. Someone must be altering packets along the way.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#229

Post by noncoml »

pclmulqdq wrote: post #214
tyu2 wrote: post #160

Earlier quoted context omitted.

Hardware that forwards packets usually forwards IP packets, it doesn't care about TCP checksums and doesn't recalculate them. TCP stack in your OS is the one that does that. It has to be a special kind of hardware that does much deeper packet inspection (DPI) to recalculate TCP checksums, usually used for spying, throttling, censorship, injecting ads, injecting exploits, etc., but not merely routing/forwarding packet…

Switches (L2 devices) recalculate Ethernet CRCs (L2 error detection codes), and routers tend to calculate TCP/UDP checksums (L3 error detection codes) and everything below. I've seen exactly this issue with switches that have bad RAM before, and I assume that they just have a router with bad RAM (and without ECC RAM, which looks embarrassing).

To my knowledge, from working on an actual software router, a router will only touch the TTL and recalculate the IP header checksum.

There is no reason for it to touch the TCP header.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#230

Post by derivagral »

astrange wrote: post #181
jedberg wrote: post #26

Earlier quoted context omitted.

NIMBYism at its finest. Cupertino did not allow cell phone towers for a very long time. The only one was an ATT tower on the top of Infinite Loop, right on the Sunnyvale border. The people who would call their provider about bad cell coverage in their house are the same people that would go to city hall and demand that no cell towers be built in the city.

It’s amusing seeing Cupertino city council transcripts about this because the people show up claiming 5G gives them cancer and the city council desperately tries to get them to use better excuses so they can approve denying it. Makes a change from the city council members usual practice of denying Vallco permits and claiming Apple employees are hiring prostitutes and molesting high school students. (I did not make th…

Are these transcripts online?
Post reply on HN