Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AT&T Fiber in the SF Bay Area is flipping bits

twitter.com

Post reply on HN
Print view View original

341–350 of 374 posts

Re: AT&T Fiber in the SF Bay Area is flipping bits

#341

Post by wccrawford »

myself248 wrote: post #280
sagarm wrote: post #163

Earlier quoted context omitted.

To be fair, the vast majority of technical support calls likely require only customer service rather than technical skills. Anyone with technical chops probably wouldn't last long in that environment.

A friend of mine used to work tech support, and said that from her perspective, a lot of the effort was trying to counter-outsmart customers who thought they were too smart. For instance: "This might just be a loose cable, can you unplug each end of it, and plug it back in?" invariably elicited an "I've already done that", or a brief pause followed by "okay, there, I did it, do you see it?". Lies, often. But: "Alrigh…

I worked for a call center and there was a girl there I didn't think was good at her job.

Then one day, she tells me that she tells people to unplug the power cord, cup the end in their hand, and then plug it back in.

Suddenly, I really liked her. It's a genius move that makes them think they've done something obscure, but she really just wanted them to actually check the cable.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#342

Post by shpx »

p1necone wrote: post #105
cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…

https://xkcd.com/806/

Re: AT&T Fiber in the SF Bay Area is flipping bits

#343

Post by satisfaction »

I don't have fiber but I do spend slightly more on business class internet, they have tech support that is an order of magnitude more helpful and knowledgeable than normal residential internet. With business class you can get real static IPs and better support SLAs.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#344

Post by wccrawford »

Nextgrid wrote: post #126
p1necone wrote: post #105

Earlier quoted context omitted.

All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…

The problem is that the idiots the hire to do their "technical" support have zero skill (nor motivation to learn) to assess whether it's a 5% problem or not, and the majority of end-users aren't capable to answer that question either, nor that they are incentivized to answer truthfully. The solution could be a priority support tier where you pay upfront for an hour of a real network engineer's time (decently compensa…

I worked for a computer support once, and just as I was hired, they went from (IIRC) 6 weeks of training to only 2 weeks. Nobody came through that knowing any more than they started with in regards to fixing computers.

Luckily, I already knew how to fix them. I found the job to be a cake walk and quite liked helping people. But I had to listen to people around me fumble through it.

It was frustrating at the time, but my favorite thing that happened was that I was admonished twice for having average call times that were too low. To them, that's a warning that someone is just getting people off the line without fixing their problems.

They monitored calls and they said I'd never received a complaint, but the system would keep flagging me for low call times so I had to artificially raise them. They suggested that I have a conversation with the clients.

I didn't, and I didn't stay there much longer, but it was quite a crazy situation. But I also felt much less pressure to handle calls quickly after that, too, which was nice.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#345

Post by joshspankit »

p1necone wrote: post #105
cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…

100% you’re on point here, but there’s one problem that happens when they do add “is this a 5% problem”: eventually (and I’d say pretty quickly), the public gets wind of “if I say the right things to get marked as a 5% problem, I get an automatic escalation to someone who knows more.”, and suddenly you get a big chunk of level 1 calls in to the upper tiers.

For evidence, see: “Retention departments give you the best rates if you threaten to cancel” (which then caused companies to have to rename the retention departments and change the policies.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#346

Post by ComputerGuru »

jlgaddis wrote: post #268
Rapzid wrote: post #232

Earlier quoted context omitted.

I have ATT fiber in Texas and was having issues recently, probably for a few days, where DNS props would just fail(I use Google's DNS), huge pauses in page loads with the occasional just doesn't. Happened over the long weekend IIRC and was sporadic enough that I didn't look into it further. I thought it had largely cleared, but now I'm wondering about some ongoing page load pauses..

> ... where DNS props would just fail ... What's a "DNS prop"?

Propagation, but I’m not entirely sure it makes sense in this context.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#347

Post by Rychard »

stevencorona wrote: post #274
exged wrote: post #249

Earlier quoted context omitted.

If you have some time, you can MITM the 802.1x auth packets [1] and use a less crappy router. I run this with a VyOS router and the same 5268ac that you have, but it works with things like Ubiquiti routers too. The only catch is you need three NICs on your router, but a cheap USB 10/100 one will do for the port that connects to the 5268ac. Another option is getting the 802.1x certificate out of a hacked router, but i…

Huge bummer but the next generation of ATT routers with onboard ONT don’t work this bypass :(

Do you know the model numbers and/or have any other information about these new routers?

I'm currently using eap_proxy with my BGW210, and it's been a huge improvement, but I fear the day the device needs to be replaced with a newer model.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#348

Post by myself248 »

thrower123 wrote: post #340
Nextgrid wrote: post #126

Earlier quoted context omitted.

The problem is that the idiots the hire to do their "technical" support have zero skill (nor motivation to learn) to assess whether it's a 5% problem or not, and the majority of end-users aren't capable to answer that question either, nor that they are incentivized to answer truthfully. The solution could be a priority support tier where you pay upfront for an hour of a real network engineer's time (decently compensa…

A huge percentage of companies have their help desks and customer support outsourced to TCS or Cognizant or Accenture, etc. Everything they do is driven by metrics, and their contracts are written around KPIs like maintaining a ludicrously short average time to answer, short handle times, and open/closed ticket ratios. If they do not hit these metrics, then the outsourcing company owes service credits. The incentives…

This blows my mind, because why would a customer come back when they're treated like that? Oh right, because they don't have anywhere else to go.

For the last several years, I've gotten my mobile service through an MVNO named Ting. On the rare occasion that I've needed to call support, there's no IVR, just a human who typically answers on the second ring. They speak native English, and have never failed to solve my problem either immediately or with one prompt escalation.

They're so jarringly competent I wonder how they still exist, if being obnoxiously incompetent is apparently a business requirement.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#349

Post by spearo77 »

cosmie wrote: post #304
comboy wrote: post #253

Earlier quoted context omitted.

Which is then a problem when you try to explain them that yes, you are sure the issue is with their service and not your setup. But what's your reason to be going such lengths instead of just plugging UDM into their router? Unless it was done for the fun of it which is fine and understandable.

> But what's your reason to be going such lengths instead of just plugging UDM into their router? While you can do this and things will generally work, AT&T restricts all of their residential gateways from operating in a true passthrough/bridge mode to another router. So you end with double NAT and all the joys that entails (such as [1]). There are also a number of other issues that have been associated with operatin…

Yes.. what he said.

But my main reason is actually the gigantic size of the residential gateway box. I mounted the ONT, UDM pro and PoE switch on a wall in a closet and the RG just took up too much space.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#350

Post by Bluecobra »

maerF0x0 wrote: post #159

Not that I have any specific evidence this is the case, but it wouldnt surprise me to later learn this is an attack on encryption by the NSA. AT&T in San Francisco is ground 0 for some of their bulk collection. After The Times disclosed the Bush administration’s warrantless wiretapping program in December 2005, plaintiffs began trying to sue AT&T and the N.S.A. In a 2006 lawsuit, a retired AT&T technician named Mark…

I would think that if that room did/still exists they would most likely use passive optical taps, not interfere with transit traffic. Optical taps simply mirror a fiber optic signal one way to a tap aggregation switch/packet capture devices. They would also be mostly undetectable, you will just see a reduced light level.
Post reply on HN