Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AT&T Fiber in the SF Bay Area is flipping bits

twitter.com

Post reply on HN
Print view View original

281–290 of 374 posts

Re: AT&T Fiber in the SF Bay Area is flipping bits

#281

Post by miahi »

francoisLabonte wrote: post #29

From my professional experience of programming and debugging networking equipment, this could be a switch/router with a buffer with bad memory (stuck bit maybe). The better chips have CRC/Parity/ECC to cover such issues but there are always those magical choke points where the past CRC is tossed and the new one is generated that can leave a gaping hole. The tricky part is how often is this bad memory buffer used... I…

This kind of incident happened to me in a system that was supposed to have high availability. We had failovers for hardware, but it seems that a network device that was supposed to have HA (and was set up to pass the functionality to another device in case of failure) did not have ECC memory. One memory bit got stuck at 0 and the event was not detected at network level, as the data was repacked with a "clean" CRC. For some reason the packet headers were not affected by this, maybe because they were kept in a separate memory zone or because of memory alignment. So the device did not report any kind of suspicious activity, no errors in its statistics.

On the application side the effects were quite bad, as the data was mainly XML and, depending on where the bit was flipped, it could impact the data or the XML structure. The data had its own CRC/hash, so the packets were cleanly rejected by the application. Unfortunately the XML library from the message queue engine and the ESB we were using did not like at all when the bit flipping occurred in the XML tags (it seems fuzzing tests were not done at that point) so the message processing got stuck and we kept getting bad messages in the queues. Even worse, the queues could not be cleaned with the normal procedures because the application wanted to first display info about the messages inside - and that failed.

The network debug was non-trivial because of that header consistency - the network devices did not report any kind of packet issues, so we had to sniff the different network segments to identify the culprit. From the application point of view, we had to delete the whole message queue storage to get rid of the bad messages, and let the application handle the rest (luckily it was designed with eventual consistency and self-healing).

Re: AT&T Fiber in the SF Bay Area is flipping bits

#283

Post by mustknow2201 »

myself248 wrote: post #265
toast0 wrote: post #222

Earlier quoted context omitted.

> If there's anyone out there designing tech support procedures, you should add an "is this a 5% problem?" question to whatever checklist you give to support staff. When I was the engineer customer service escalated to, I was damn sure to thank them every time they escalated something. Even the one guy who escalated all the things I'd roll my eyes about in private. At least he was making sure the escalation path work…

Ex-phone-company here. (Is this the party to whom I am speaking?) I was in installation, but hung out with a lot of the ops crew, and they LOVED interesting problems. The trouble was getting such problems to the ops people in the first place. Good people, bad process. The most memorable one: Customer service had been getting calls all morning with a peculiar complaint: A customer's phone would ring, and when they ans…

Cool story, thanks! Perhaps you can solve a mystery phone hiccup that happened to me a few years ago? I called a friend (mobile to mobile if it matters) and, from memory, about 20 minutes into this call I get disconnected, _but_ I instantly end up on a call with an elderly stranger instead, who seemed pretty irritated she was now on the phone with me. I was surprised enough that she hung up before I could form a coherent sentence to explain what had happened so I've no idea if she was trying to ring someone or if the same thing happened to her or if she'd dialed my number by accident. From what I remember it seemed like she was also already mid-conversation as well though.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#284

Post by gigatexal »

myself248 wrote: post #265
toast0 wrote: post #222

Earlier quoted context omitted.

> If there's anyone out there designing tech support procedures, you should add an "is this a 5% problem?" question to whatever checklist you give to support staff. When I was the engineer customer service escalated to, I was damn sure to thank them every time they escalated something. Even the one guy who escalated all the things I'd roll my eyes about in private. At least he was making sure the escalation path work…

Ex-phone-company here. (Is this the party to whom I am speaking?) I was in installation, but hung out with a lot of the ops crew, and they LOVED interesting problems. The trouble was getting such problems to the ops people in the first place. Good people, bad process. The most memorable one: Customer service had been getting calls all morning with a peculiar complaint: A customer's phone would ring, and when they ans…

What’s fascinating read! Gotta error check my scripts.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#285

Post by 013a »

AaronFriel wrote: post #93

Trying to explain this issue to AT&T support is like trying to convince a doctor you're the only person on earth with a particular disease. Even explaining the issue is hard. It's not an outage, my internet isn't out , it's intermittently wrong . The phone support agents aren't prepared for this, and I can't find any way to escalate or speak to a network engineer. I feel like if I spoke to the right engineer, there'd…

Have you seen Parks & Rec, and remember that scene in a Home Depot where an associate walks up to Ron, asks him if he needs help with a project, and Ron responds "I know more than you"?

I've pulled a variation of that on CSRs at least once, and surprisingly, it can work. Just be cordial, preempt the typical IT Support stuff they always ask, DO NOT say its intermittent (initially, to the front line CSR; if given a chance to expand the issue after escalation, then add that bit), and get technical ASAP (it doesn't hurt to throw in some parallel industry jargon). Basically, build a case where even the information you're giving them is beyond a first-line CSR playbook, and they have to escalate.

"Hi there; I've been observing some erroneous TCP packet bit flipping on HTTP requests which route through one of AT&T's data center in Oakland. I've tried restarting my computer, I'm seeing the same thing on my phone, and I actually swapped my router out for a spare one I have, but its still an issue."

(that last sentence exhausts literally every playbook a front-line CSR has. it sounds so easy, right? there are four variables in any front-line CSR diagnostic equation: their network, your router, wifi/ethernet, and the endpoint. you just crossed off three of the four variables in one sentence).

(Wait, a data center in Oakland? How do you know this? You can tracert a bad request and geolocate the first IP outside your network, but, lets be realistic: You don't. You're fronting; demonstrating knowledge that a front-line CSR can't disprove. You may think this is misleading to whoever this gets escalated to, but it isn't; their tools are FAR more advanced than yours, and they're used to 99% of customers being incorrect idiots, so they're going to be validating and reconfirming every word you say anyway.)

Ron's Parks & Rec example above is crass. But here's the magic bit: frontline CSRs generally look for an excuse to escalate, you just need to give them enough CYA to check their job as done, and the higher tier CSRs/network engineers will love you for actually knowing what you're talking about. Its a win-win; be cordial, be forceful, strut what you know.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#286

Post by nikanj »

p1necone wrote: post #105
cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…

Many companies have done the math, and realized they make more money if they just let the 5% customers leave.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#287

Post by hevisko »

danceparty wrote: post #95

AT&T runs transparent proxies on their network that have been known to intercept and mangle tls, and directly manipulate tcp packets on the wire. Take a tcpdump capture on both server/client side and compare the handshake/tcp headers/negotiated window size. Often you see the server sending one thing, but the client sees another

question: could people confirm the ssl certificate chains being used by the server, and reported by the client?

Re: AT&T Fiber in the SF Bay Area is flipping bits

#288

Post by zanny »

013a wrote: post #285
AaronFriel wrote: post #93

Trying to explain this issue to AT&T support is like trying to convince a doctor you're the only person on earth with a particular disease. Even explaining the issue is hard. It's not an outage, my internet isn't out , it's intermittently wrong . The phone support agents aren't prepared for this, and I can't find any way to escalate or speak to a network engineer. I feel like if I spoke to the right engineer, there'd…

Have you seen Parks & Rec, and remember that scene in a Home Depot where an associate walks up to Ron, asks him if he needs help with a project, and Ron responds "I know more than you"? I've pulled a variation of that on CSRs at least once, and surprisingly, it can work. Just be cordial, preempt the typical IT Support stuff they always ask, DO NOT say its intermittent (initially, to the front line CSR; if given a cha…

I had something like this happen on an even simpler level this last week. I got a Chase credit card but when I initially did the signup called my brother to ask him if he wanted to be on the account and it timed the sign up session out past the account creation but before finalization.

I got the card eventually but now I cannot create an online account with it. I called Chase, got transferred 5 times, and then told I would need to go to a physical bank to verify my identity? to create an account. Absolutely not one of them had any clue what "a broken account exists associated with this card in your database, I can guarantee it, forward me to your technical support team" but thats all above a bank reps pay grade.

The nearest Chase bank is 1.5 hours away, by the way. Probably just going to cancel the card after cashing out the sign up bonus.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#289

Post by ummonk »

boulos wrote: post #130
secalex wrote: post #76

You can contribute to our attempts to find the bad router card here: https://twitter.com/alexstamos/status/1336099461622157312 Almost certainly 12.242.117.22

Indeed. My mobile.twitter.com traceroute prefers going through that path, as does en.wikipedia.org (both of which have sucked for me) while a Google route (to 172.217.6.78) hops through 12.122.149.186.

Yup, I'd been having issues with twitter, wikipedia, and sometimes duck duck go but never google.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#290

Post by X-Istence »

SV_BubbleTime wrote: post #241
tialaramex wrote: post #209

Earlier quoted context omitted.

> presumably because a corrupted packet in a TLS handshake is an error and the connection dies Not just in the handshake, TLS moves these things called TLSPlaintext records (about 16kbytes each), not only in the handshake, but also for all the actual data - and they'll always have integrity protection to ensure bad guys can't change anything. TLS can't know the difference between a bad guy tampering with data and you…

Help me understand, are you saying only HTTP bits are being flipped? Because yea, if a HTTPS bit was flipped the whole packet dies. So is this issue blowing up all sorts of traffic everywhere?

Yes, all bits are being flipped. TLS connections drop because the message can't be authenticated, HTTP or other plaintext protocols will continue on with bad data.

So yes, it is blowing up all sorts of traffic everywhere. You just don't notice when it is plaintext.

Post reply on HN