Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AT&T Fiber in the SF Bay Area is flipping bits

twitter.com

Post reply on HN
Print view View original

241–250 of 374 posts

Re: AT&T Fiber in the SF Bay Area is flipping bits

#241

Post by SV_BubbleTime »

tialaramex wrote: post #209
cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

> presumably because a corrupted packet in a TLS handshake is an error and the connection dies Not just in the handshake, TLS moves these things called TLSPlaintext records (about 16kbytes each), not only in the handshake, but also for all the actual data - and they'll always have integrity protection to ensure bad guys can't change anything. TLS can't know the difference between a bad guy tampering with data and you…

Help me understand, are you saying only HTTP bits are being flipped? Because yea, if a HTTPS bit was flipped the whole packet dies. So is this issue blowing up all sorts of traffic everywhere?

Re: AT&T Fiber in the SF Bay Area is flipping bits

#242

Post by shuckles »

The same day there is an enormous prognostication thread on the future of the Bay Area, a bunch of nerds on Twitter observe, test, diagnose, and root cause a local fiber network issue. I can only think of a few places in the world where this may happen: https://twitter.com/bd/status/1336110887145361410

Re: AT&T Fiber in the SF Bay Area is flipping bits

#243

Post by GauntletWizard »

saagarjha wrote: post #172
GauntletWizard wrote: post #142

Earlier quoted context omitted.

Who decides if it's the ISP's problem? What if it's "Both"? About 20 years ago I got escalated to high-tier Comcast support for an issue that turned out to be a little of A, a little of B: Comcast (Might have been @Home, based on the timing) required that your MAC address be whitelisted as part of their onboarding process. Early home routers had a "MAC Address Clone" feature for precisely this reason. At some point,…

How did you even manage to figure this out? I’ve never gotten anyone on the phone who could possibly help in a situation like this.

Wireshark and escalation to a competent tech. I believe they saw weird traffic from their DHCP server, and we were able to attach an ethernet hub (Not switch, a 10-Base-T Hub that repeated the signal on each port) along with a laptop that was running Ethereal (Before the name changed! How long ago that was now) and see the arp packets fighting.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#244

Post by _rs »

warent wrote: post #122
dudul wrote: post #120

Earlier quoted context omitted.

Wait so you have your main cable plugged into their modem, their router plugged into their modem, and then your own router plugged into their router?

Yes exactly. Their router has a LAN with my router as the only other device, which it's bridged with, and then my router has the true home LAN. A weird side effect of this is that I'm not using the 192.168.x.x range like usual (because that's what theirs is using), but instead the 10.0.x.x range

So are you bridged then or is it really a double nat?

Re: AT&T Fiber in the SF Bay Area is flipping bits

#245

Post by kmeisthax »

jasomill wrote: post #197
cbhl wrote: post #14

When AT&T first did their Fiber rollout in SF, one of the things I remember was they charged you $10 extra if you didn't want them to MITM all your connections to insert JavaScript pointing to their own ads. They rolled this back when folks complained, but I wonder if the relevant infrastructure is still sitting around and mangling packets. I would probably expect this to be some network card or cable or connector is…

What I always wondered about this is, unless AT&T has permission from the copyright holder of each and every Web page so modified to distribute the resulting derivative work, how is this practice not criminal copyright infringement?

For the same reason why ISPs are generally immunized from contributory liability that they would otherwise be completely buried in.

Now, if you had written JavaScript to detect and remove these ads, and they went around that, then you might be able to construct a DMCA 1201 claim and sue the ISP for circumventing what is legally considered DRM. Yes, JavaScript can be legally protected DRM. The law doesn't say it has to be good DRM, it just has to have the effect of controlling access to a copyrighted work. And the safe harbors the DMCA provides ISPs wouldn't protect them in this case.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#246

Post by garmaine »

noncoml wrote: post #229
pclmulqdq wrote: post #214

Earlier quoted context omitted.

Switches (L2 devices) recalculate Ethernet CRCs (L2 error detection codes), and routers tend to calculate TCP/UDP checksums (L3 error detection codes) and everything below. I've seen exactly this issue with switches that have bad RAM before, and I assume that they just have a router with bad RAM (and without ECC RAM, which looks embarrassing).

To my knowledge, from working on an actual software router, a router will only touch the TTL and recalculate the IP header checksum. There is no reason for it to touch the TCP header.

Agreed there is no logical reason for it to touch the TCP header.

And yet, an unfortunate number of L2 switches do exactly that. :(

Re: AT&T Fiber in the SF Bay Area is flipping bits

#247

Post by _rs »

cbhl wrote: post #14

When AT&T first did their Fiber rollout in SF, one of the things I remember was they charged you $10 extra if you didn't want them to MITM all your connections to insert JavaScript pointing to their own ads. They rolled this back when folks complained, but I wonder if the relevant infrastructure is still sitting around and mangling packets. I would probably expect this to be some network card or cable or connector is…

As someone who doesn’t have ATT as an option here, clearly they can’t be that blatant about it?! How did they spin this $10 fee?

Re: AT&T Fiber in the SF Bay Area is flipping bits

#248

Post by bakul »

Scoundreller wrote: post #200
bakul wrote: post #143

Earlier quoted context omitted.

The same here. It is always the 0x08 bit. (p-x, d-l, g-o etc.).

I wonder if there’s some 64 line I/O somewhere with a couple traces bridged together, where: 0 and 0 are still 0 and 0, 1 and 1 are still 1 and 1, but 0 and 1 become 1 and 1 (or 0&0) and 1 and 0 become 1 and 1 (or 0&0). I remember seeing this wackiness when I bridged two address lines on an EEPROM with the tiniest amount of solder. RoHS/tin whiskers strikes again?

This is unlikely, given how many people are seeing this. And now the problem has gone away for me!

Re: AT&T Fiber in the SF Bay Area is flipping bits

#249

Post by exged »

Shank wrote: post #233
m463 wrote: post #223

Earlier quoted context omitted.

I had a ginormous AT&T router/modem (pace 5268ac) with a set of static ip addresses and a few times, AT&T just stopped routing traffic to it. It had happened before and then magically fixed itself a few days later. One time I had a week of outage with AT&T basically said the problem was on my side. They could ping the modem, and then punted. I had several truck rolls. The techs were really nice guys, but were basical…

The 5268ac pace router is the worst ISP provided router I've ever had, and I've been an Xfinity/Comcast customer, and I've even had a connection in Wyoming. I detailed my experience with it in a review of a third-party router, and found numerous issues along the way [0]. My favorite is that DMZ+ mode, which is what they offer instead of a traditional DMZ mode, just has some weird MTU issue that leads git and other se…

If you have some time, you can MITM the 802.1x auth packets [1] and use a less crappy router. I run this with a VyOS router and the same 5268ac that you have, but it works with things like Ubiquiti routers too. The only catch is you need three NICs on your router, but a cheap USB 10/100 one will do for the port that connects to the 5268ac.

Another option is getting the 802.1x certificate out of a hacked router, but it's not possible as far as I know on the 5268ac. You could buy a hackable ATT router but they're not cheap. Some sellers even sell the key by itself.

Mysteriously, doing this fixed an issue I previously had where SSHing into AWS would fail.

[1] https://github.com/jaysoffian/eap_proxy

Re: AT&T Fiber in the SF Bay Area is flipping bits

#250

Post by rgj »

duxup wrote: post #220
p1necone wrote: post #105

Earlier quoted context omitted.

All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…

Way back in the day I worked on equipment that straddled telco circuits. T1s, E1s, DS3s, OC whatever. Companies paying big money for those circuits. Anyway I was told on more than one occasion by different telcos that the standard operating procedure for many techs was to take the call, do nothing, and call back 20 minutes later and ask if it looked better because ... often enough it did.

When I started my job as an IT director 10 years ago I was in the customer support room and there was one guy who was known for solving all the hard problems. I was standing behind him when he was talking a call. He patiently listened to the client, then loudly typed random stuff on his keyboard for twenty seconds or so, making sure the client could hear the frantic typing, sighed, and then asked “Is it better now?”

It always was.

Post reply on HN