AT&T Fiber in the SF Bay Area is flipping bits • Hacker News Board

Re: AT&T Fiber in the SF Bay Area is flipping bits

#271

Post by pakl » Tue, Dec 08, 2020, 6:50 AM UTC

p1necone wrote: post #105

cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…

That’s why you need to sign up for CSA Pre™. Get preauthorized for instant escalation on customer support calls.

All you have to do is answer a form with questions like: Do you know how to plug in a computer? Do you know where the power switches are on your devices? etc.

CSA Pre™ is valid for 5 years; you can initiate the renewal process to do 6 months before expiry.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#272

Post by 013a » Tue, Dec 08, 2020, 6:52 AM UTC

gamegoblin wrote: post #127

AaronFriel wrote: post #93

Trying to explain this issue to AT&T support is like trying to convince a doctor you're the only person on earth with a particular disease. Even explaining the issue is hard. It's not an outage, my internet isn't out , it's intermittently wrong . The phone support agents aren't prepared for this, and I can't find any way to escalate or speak to a network engineer. I feel like if I spoke to the right engineer, there'd…

When I first moved to Seattle there was a great local ISP called CondoInternet that mainly specialized in high-density downtown buildings. I was once having some packet loss issues and called their support line. I assume the company was really small at the time, because the guy who answered the phone was clearly a network engineer who knew the system inside and out. I read him a couple of traceroutes over the phone a…

I had a similar, local fiber ISP in a high density midwest apartment. Truly fantastic support. The network engineer came out to set me up, and we got to talking about our backgrounds (I worked in IT support while attending a nearby tech-focused university), he invites me to check out their building-wide switch closet down the hall with all the cool gear, then says "yeah we limit everyone to 100/100, but I'll flag your account for 250, just don't go overboard eh"

Scale ruins most things, unfortunately.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#273

Post by ornornor » Tue, Dec 08, 2020, 6:54 AM UTC

pakl wrote: post #271

p1necone wrote: post #105

Earlier quoted context omitted.

All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…

That’s why you need to sign up for CSA Pre™. Get preauthorized for instant escalation on customer support calls. All you have to do is answer a form with questions like: Do you know how to plug in a computer? Do you know where the power switches are on your devices? etc. CSA Pre™ is valid for 5 years; you can initiate the renewal process to do 6 months before expiry.

And it’s a steal at 99.90/month with a 24 months contract!

Re: AT&T Fiber in the SF Bay Area is flipping bits

#274

Post by stevencorona » Tue, Dec 08, 2020, 6:55 AM UTC

exged wrote: post #249

Shank wrote: post #233

Earlier quoted context omitted.

The 5268ac pace router is the worst ISP provided router I've ever had, and I've been an Xfinity/Comcast customer, and I've even had a connection in Wyoming. I detailed my experience with it in a review of a third-party router, and found numerous issues along the way [0]. My favorite is that DMZ+ mode, which is what they offer instead of a traditional DMZ mode, just has some weird MTU issue that leads git and other se…

If you have some time, you can MITM the 802.1x auth packets [1] and use a less crappy router. I run this with a VyOS router and the same 5268ac that you have, but it works with things like Ubiquiti routers too. The only catch is you need three NICs on your router, but a cheap USB 10/100 one will do for the port that connects to the 5268ac. Another option is getting the 802.1x certificate out of a hacked router, but i…

Huge bummer but the next generation of ATT routers with onboard ONT don’t work this bypass :(

Re: AT&T Fiber in the SF Bay Area is flipping bits

#275

Post by ornornor » Tue, Dec 08, 2020, 6:58 AM UTC

myself248 wrote: post #265

toast0 wrote: post #222

Earlier quoted context omitted.

> If there's anyone out there designing tech support procedures, you should add an "is this a 5% problem?" question to whatever checklist you give to support staff. When I was the engineer customer service escalated to, I was damn sure to thank them every time they escalated something. Even the one guy who escalated all the things I'd roll my eyes about in private. At least he was making sure the escalation path work…

Ex-phone-company here. (Is this the party to whom I am speaking?) I was in installation, but hung out with a lot of the ops crew, and they LOVED interesting problems. The trouble was getting such problems to the ops people in the first place. Good people, bad process. The most memorable one: Customer service had been getting calls all morning with a peculiar complaint: A customer's phone would ring, and when they ans…

Cool story, thanks for sharing!

Re: AT&T Fiber in the SF Bay Area is flipping bits

#276

Post by ericbarrett » Tue, Dec 08, 2020, 7:00 AM UTC

cranekam wrote: post #261

willis936 wrote: post #228

Earlier quoted context omitted.

I don’t have numbers off hand, but from a feeling I have from memory, I would think it is extremely unlikely that TCP checksums are consistently failing to trigger retransmission. Someone must be altering packets along the way.

TCP checksums are notoriously weak: https://www.evanjones.ca/tcp-checksums.html With enough packets passing the dodgy RAM a noticeable number will manage to get mangled in such a way that the checksum is still correct.

Checksums are also often recomputed on transit if the packet is intercepted, e.g. to limit TTL, unflag odd/unused TCP features, that kind of ISP-ish preening. So if it was a software error or even a hardware problem in the right (wrong) spot it’s possible to get this kind of corruption without retransmits.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#277

Post by cbozeman » Tue, Dec 08, 2020, 7:03 AM UTC

runawaybottle wrote: post #138

p1necone wrote: post #105

Earlier quoted context omitted.

All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…

Anecdote from my past: I used to play counter strike/Starcraft in my middle school years. I pretty much figured out I had consistent packet loss with a simple ping test. I was on the phone with Time Warner every other day for months. They kept sending the regular technicians, at one point ripping out all of my cable wires in the house to see if it fixed the problem. Nothing worked, I kept calling, at this point I had…

The real problem is that it shouldn't take this much bullshit and rigmarole from the very beginning.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#278

Post by BlueTankEngine » Tue, Dec 08, 2020, 7:09 AM UTC

myself248 wrote: post #265

toast0 wrote: post #222

Earlier quoted context omitted.

> If there's anyone out there designing tech support procedures, you should add an "is this a 5% problem?" question to whatever checklist you give to support staff. When I was the engineer customer service escalated to, I was damn sure to thank them every time they escalated something. Even the one guy who escalated all the things I'd roll my eyes about in private. At least he was making sure the escalation path work…

Ex-phone-company here. (Is this the party to whom I am speaking?) I was in installation, but hung out with a lot of the ops crew, and they LOVED interesting problems. The trouble was getting such problems to the ops people in the first place. Good people, bad process. The most memorable one: Customer service had been getting calls all morning with a peculiar complaint: A customer's phone would ring, and when they ans…

This is a super cool story, thank you so much for taking the time to type this out!

Re: AT&T Fiber in the SF Bay Area is flipping bits

#279

Post by sumosudo » Tue, Dec 08, 2020, 7:11 AM UTC

Well I flipped the bit right back at them!

But seriously now, have they tried turning it off and on again?

Re: AT&T Fiber in the SF Bay Area is flipping bits

#280

Post by myself248 » Tue, Dec 08, 2020, 7:15 AM UTC

sagarm wrote: post #163

Nextgrid wrote: post #126

Earlier quoted context omitted.

The problem is that the idiots the hire to do their "technical" support have zero skill (nor motivation to learn) to assess whether it's a 5% problem or not, and the majority of end-users aren't capable to answer that question either, nor that they are incentivized to answer truthfully. The solution could be a priority support tier where you pay upfront for an hour of a real network engineer's time (decently compensa…

To be fair, the vast majority of technical support calls likely require only customer service rather than technical skills. Anyone with technical chops probably wouldn't last long in that environment.

A friend of mine used to work tech support, and said that from her perspective, a lot of the effort was trying to counter-outsmart customers who thought they were too smart.

For instance: "This might just be a loose cable, can you unplug each end of it, and plug it back in?" invariably elicited an "I've already done that", or a brief pause followed by "okay, there, I did it, do you see it?". Lies, often.

But: "Alright, I want you to try turning the cable around for me, yeah swap it end for end. Sometimes you get a connector that just doesn't fit quite right, but it works the other way around and it's faster than sending a tech to replace it", would often get a startled "Oh! I hadn't thought of that, one moment..." and then the customer actually DOES unplug the thing, and what do you know, they click it in properly this time.