Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AT&T Fiber in the SF Bay Area is flipping bits

twitter.com

Post reply on HN
Print view View original

331–340 of 374 posts

Re: AT&T Fiber in the SF Bay Area is flipping bits

#331

Post by mjevans »

nikanj wrote: post #286
p1necone wrote: post #105

Earlier quoted context omitted.

All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…

Many companies have done the math, and realized they make more money if they just let the 5% customers leave.

It's even worse; the customers have no real choice of anyone else to leave _to_.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#332

Post by coldpie »

porknubbins wrote: post #157
p1necone wrote: post #105

Earlier quoted context omitted.

All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…

And conversely its always surprising and disarming when you call a company and actually get through to a knowledgable employee. I was so surprised to hear “thats a firmware bug we know about and there is no update yet” about my router issue that I forgot to be mad at the company for not caring my router is broken.

A few years back, I called up our local newspaper to start a subscription. Called the number on the website, and a real human person answered the phone. I was so surprised that it wasn't at least an initial phone tree that I actually stumbled and had to apologize and explain myself.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#333

Post by simias »

bcrl wrote: post #252
francoisLabonte wrote: post #29

From my professional experience of programming and debugging networking equipment, this could be a switch/router with a buffer with bad memory (stuck bit maybe). The better chips have CRC/Parity/ECC to cover such issues but there are always those magical choke points where the past CRC is tossed and the new one is generated that can leave a gaping hole. The tricky part is how often is this bad memory buffer used... I…

Hardware designers basically started making bad decisions on this issue around the time that VLAN tagging was introduced, as well as harware forwarding of IP packets. When VLAN tags are inserted or removed, the CRC of a packet needs to be adjusted to reflect the inserted, removed and/or modified bytes from the VLAN header. Additionally Both the CRC and IP checksum of a packet needs to be adjusted when TTL is decremen…

I'm not a hardware designer but I routinely deal with low level networking shenanigans and I must admit that I never considered that it would be possible to update a CRC without recomputing it fully (unless you were just appending data of course).

For people like me who aren't smart enough to figure it on their own, this stackexchange answer seem to explain how it's done: https://cs.stackexchange.com/questions/92279/can-one-quickly...

Re: AT&T Fiber in the SF Bay Area is flipping bits

#334

Post by kalleboo »

_rs wrote: post #247
cbhl wrote: post #14

When AT&T first did their Fiber rollout in SF, one of the things I remember was they charged you $10 extra if you didn't want them to MITM all your connections to insert JavaScript pointing to their own ads. They rolled this back when folks complained, but I wonder if the relevant infrastructure is still sitting around and mangling packets. I would probably expect this to be some network card or cable or connector is…

As someone who doesn’t have ATT as an option here, clearly they can’t be that blatant about it?! How did they spin this $10 fee?

https://arstechnica.com/information-technology/2016/09/att-t...

Re: AT&T Fiber in the SF Bay Area is flipping bits

#335

Post by derangedHorse »

14u2c wrote: post #312
m463 wrote: post #223

Earlier quoted context omitted.

I had a ginormous AT&T router/modem (pace 5268ac) with a set of static ip addresses and a few times, AT&T just stopped routing traffic to it. It had happened before and then magically fixed itself a few days later. One time I had a week of outage with AT&T basically said the problem was on my side. They could ping the modem, and then punted. I had several truck rolls. The techs were really nice guys, but were basical…

That is absurd that AT&T requires the use of a rented gateway for U-verse. I've never had an issue with a another provider refusing to support off the self hardware before, including with a DSL provider, multiple cable companies, and FiOS (Ethernet on ONT).

AT&T sucks, congress should be requesting their leaders in for a congressional hearing along with Zuckerberg lol

Re: AT&T Fiber in the SF Bay Area is flipping bits

#337

Post by stronglikedan »

013a wrote: post #285
AaronFriel wrote: post #93

Trying to explain this issue to AT&T support is like trying to convince a doctor you're the only person on earth with a particular disease. Even explaining the issue is hard. It's not an outage, my internet isn't out , it's intermittently wrong . The phone support agents aren't prepared for this, and I can't find any way to escalate or speak to a network engineer. I feel like if I spoke to the right engineer, there'd…

Have you seen Parks & Rec, and remember that scene in a Home Depot where an associate walks up to Ron, asks him if he needs help with a project, and Ron responds "I know more than you"? I've pulled a variation of that on CSRs at least once, and surprisingly, it can work. Just be cordial, preempt the typical IT Support stuff they always ask, DO NOT say its intermittent (initially, to the front line CSR; if given a cha…

> I've tried restarting my computer, I'm seeing the same thing on my phone, and I actually swapped my router out for a spare one I have, but its still an issue.

"Ok sir, please click the start button, then the power button, and finally click the restart button to restart your computer..." (and they refuse to budge until you've swapped out your router yet again, because you didn't do all that while you were on the phone with them)

Re: AT&T Fiber in the SF Bay Area is flipping bits

#338

Post by gspr »

gamegoblin wrote: post #127
AaronFriel wrote: post #93

Trying to explain this issue to AT&T support is like trying to convince a doctor you're the only person on earth with a particular disease. Even explaining the issue is hard. It's not an outage, my internet isn't out , it's intermittently wrong . The phone support agents aren't prepared for this, and I can't find any way to escalate or speak to a network engineer. I feel like if I spoke to the right engineer, there'd…

When I first moved to Seattle there was a great local ISP called CondoInternet that mainly specialized in high-density downtown buildings. I was once having some packet loss issues and called their support line. I assume the company was really small at the time, because the guy who answered the phone was clearly a network engineer who knew the system inside and out. I read him a couple of traceroutes over the phone a…

I had a similarly stellar experience with Init7 in Switzerland. The person who answered the phone wasn't a network engineer, but immediately passed me on to one within 10 seconds of our conversation starting.

For me it was night and day after previous having had a municipality-run absolutely garbage ISP previously (CityCable, for anyone considering them, stay far far away). Init7 might be a tad more expensive than most, but the service is solid.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#339

Post by Unklejoe »

My first thought was: how come the frame isn't just dropped due to a bad checksum?

Then I remembered a piece of Ethernet test hardware I worked with. It had a bad SRAM chip or something which would result in all of the payloads having a stuck bit every 8 bytes. However, the FCS was calculated _after_ that, so it was still a valid frame.

My guess is AT&T has some hardware that's recalculating the checksums after the frame has already been corrupted.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#340

Post by thrower123 »

Nextgrid wrote: post #126
p1necone wrote: post #105

Earlier quoted context omitted.

All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…

The problem is that the idiots the hire to do their "technical" support have zero skill (nor motivation to learn) to assess whether it's a 5% problem or not, and the majority of end-users aren't capable to answer that question either, nor that they are incentivized to answer truthfully. The solution could be a priority support tier where you pay upfront for an hour of a real network engineer's time (decently compensa…

A huge percentage of companies have their help desks and customer support outsourced to TCS or Cognizant or Accenture, etc.

Everything they do is driven by metrics, and their contracts are written around KPIs like maintaining a ludicrously short average time to answer, short handle times, and open/closed ticket ratios. If they do not hit these metrics, then the outsourcing company owes service credits. The incentives do not align with making customers happy and solving their problems. Everything is geared towards deflecting users with self-help options, simple scripts for the agents, and walking the line of hitting those metrics with the fewest number of warm bodies.

It's a pretty hellish business.

Post reply on HN