AT&T Fiber in the SF Bay Area is flipping bits • Hacker News Board

Re: AT&T Fiber in the SF Bay Area is flipping bits

#291

Post by ummonk » Tue, Dec 08, 2020, 7:37 AM UTC

cameldrv wrote: post #40

cbhl wrote: post #16

Earlier quoted context omitted.

The ISPs have tried, but they get pushback from local residents whenever they try to install the necessary network boxes at intervals down the street. They had to go out of their way to hide cellular towers as streetlamps to get the 5G rollout to happen; I remember getting two or three notices for this (each from different telcos, so the one street corner has three streetlamps and two traffic lights on one corner). I…

The parts of SV that haven't buried their power lines (not too pretty to begin with) have gotten significantly uglier since all of the new cable/fiber/DSL infrastructure has gone in. There are in many cases multiple of these things on loads of poles: https://www.dslreports.com/forum/r31593060-Equip-What-is-thi...

No, it hasn't made anything uglier - you don't really notice the infrastructure unless you're specifically trying to look for something to get annoyed by.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#292

Post by cranekam » Tue, Dec 08, 2020, 7:37 AM UTC

cranekam wrote: post #269

toast0 wrote: post #224

Earlier quoted context omitted.

Or their computer isn't checking the checksum. As is apparently the case on mac os (as reported elsewhere in the thread). Another quality product by Apple ;P

There’s no proof of this. And what possible reason would macOS have for not checking the checksum? Although the checksum is weak it presumably catches at least some corrupt traffic. Do you really think Apple would just skip the TCP checksum and make its network performance less reliable when they have already implemented (or maintained if it came from BSD) the rest of a TCP/IP stack, which is vastly more complex, jus…

It also seems quite easy to verify this hypothesis with scapy:

  >>> p1 = IP(dst="192.168.mac.ip")/TCP(dport=1984,sport=20001)
  >>> p2 = IP(dst="192.168.mac.ip")/TCP(dport=1984,sport=20002)
  >>> p2.show2()
  ###[ IP ]###
    version   = 4
    ihl       = 5
    tos       = 0x0
    len       = 40
    id        = 1
    flags     =
    frag      = 0
    ttl       = 64
    proto     = tcp
    chksum    = 0xf905
    src       = 192.168.linux.ip
    dst       = 192.168.mac.ip
    \options   \
  ###[ TCP ]###
       sport     = commtact_http
       dport     = bb
       seq       = 0
       ack       = 0
       dataofs   = 5
       reserved  = 0
       flags     = S
       window    = 8192
       chksum    = 0xb836
       urgptr    = 0
       options   = []

  >>> p2[TCP].chksum = 0xb836 ^ 0x8  # mangled checksum

  >>> sr1(p1, timeout=1)
  Begin emission:
  Finished sending 1 packets.
  ....*
  Received 5 packets, got 1 answers, remaining 0 packets
  >

  >>> sr1(p2, timeout=1)
  Begin emission:
  Finished sending 1 packets.
  .....................................................................................................................................................................................
  .....................................................................................................................................................................................
  ....................................................................................................................................
  Received 494 packets, got 0 answers, remaining 1 packets

My Mac silently dropped the packet with the mangled checksum.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#293

Post by ummonk » Tue, Dec 08, 2020, 7:42 AM UTC

e40 wrote: post #78

ProfessorLayton wrote: post #47

I had AT&T via Sonic's FTTN in an effort to avoid Comcast/AT&T directly, especially since Sonic does not have data caps despite running on AT&T's network and actually respects their customer's privacy. Unfortunately my experience was so terrible and unreliable that I decided to give up and finally sign up with Comcast, and just have to mind that 1.2TB data cap. The Bay Area of all places should not have such terrible…

I was seriously considering signing up for Sonic service, and I have Comcast now. Guess I won't do that. I hesitated because I really really hate AT&T. Guess I should have trusted that feeling.

Fwiw, I have Sonic's FTTN and it has been a great experience up till this issue started last month.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#294

Post by orlp » Tue, Dec 08, 2020, 7:42 AM UTC

pakl wrote: post #271

p1necone wrote: post #105

Earlier quoted context omitted.

All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…

That’s why you need to sign up for CSA Pre™. Get preauthorized for instant escalation on customer support calls. All you have to do is answer a form with questions like: Do you know how to plug in a computer? Do you know where the power switches are on your devices? etc. CSA Pre™ is valid for 5 years; you can initiate the renewal process to do 6 months before expiry.

At this point we unironically need shibboleet.

https://xkcd.com/806/

Re: AT&T Fiber in the SF Bay Area is flipping bits

#295

Post by IntelMiner » Tue, Dec 08, 2020, 7:49 AM UTC

neelc wrote: post #194

gamegoblin wrote: post #127

Earlier quoted context omitted.

When I first moved to Seattle there was a great local ISP called CondoInternet that mainly specialized in high-density downtown buildings. I was once having some packet loss issues and called their support line. I assume the company was really small at the time, because the guy who answered the phone was clearly a network engineer who knew the system inside and out. I read him a couple of traceroutes over the phone a…

And when Wave bought them, Wave put incompetent people in charge. I moved to a Seattle suburb and had "Wave G" (post-Wave CondoInternet). I had throughput issues, first on my apartment. That got resolved via Reddit. Then the backbone is slow from time to time which persisted. Sometimes I get a Gigabit, other times I only get 10-20 Mbps. What did Wave say? Let's bring a tech out. I told them "it's a backbone issue" an…

I made the mistake of (briefly) working at Wave. I can probably shed some light on them (full disclosure: they fired me after I was sent home for vomiting in the bathroom at work during a high call volume evening)

Wave is probably one of the most breathtakingly "if it works, use it" companies I've ever seen. I'm not sure if any other ISP's truly even compare simply in the breadth of equipment they use, combined with how poorly they operate

Wave operates in parts of Washington, Oregon and California. They've mostly grown by acquiring smaller, unprofitable or mis-managed ISP's in the areas they now own.

Wave offers TV, Internet and Phone services. Unlike a typical ISP like Comcast or Frontier however they don't just offer a single or a couple methods of service delivery

For TV service, Wave offers

- Analog TV (mostly areas in California)

- Digital TV in most other areas

- TiVO and CableCARD services (they don't have the purchasing power to get boxes from companies like Motorola/Arris or Cisco/Pace)

For Telephony service. They not only offer VoIP services but in some areas even offer regular PSTN phone lines!

For internet, Wave offers its "Wave" DOCSIS 3.0 (and in many areas, still 2.0) services. They also own what was originally CondoInternet

Finding out how CondoInternet/Wave G operates was probably one of the most horrifying things I've ever seen in my years of telco work. I'll try to explain it from the ground up since it's very much a jenga tower of terribleness

Condo Internet in its inception had a very uphill battle. They wanted to target expensive Seattle condo buildings and sell a "premium" product. However in very Seattle fashion they were slapped with very indifferent, if not outright hostile actors to their plans. So they made do with what they could get

Condo Internet's services comprise a hodgepodge of VDSL2, Point to Point wireless, Fiber-Optic and MoCA. Effectively what ever they could wire into the building or appropriate for use, they did. This is why some apartments can get symmetric gigabit, while others can only get 100 megabit

MoCA was initially the most horrifying one I encountered while working there. MoCA is effectively Ethernet running over Coaxial cables. Except since Coax is a shared medium, it's just like the ethernet hubs of the 1990's all over again.

The main reason this was done as it was considered cheaper than installing an HFC node or CMTS. They didn't know how many customers they would get to switch over, so they played their cards extremely conservatively.

Apartments with MoCA configured would have a (managed) gigabit switch or two in the basement for link back to the Condo PoP. Whichever vendor was cheapest at time of purchase (Cisco, Juniper etc)

These switches would each be connected to an individual MoCA adapter, connected to one of the cable drops going to each individual apartment/floor/whatever. The field tech would then install an accompanying MoCA adapter in the customers home (simply calling it a "cable modem") and connect it to a Wave provided router (typically a TP-Link Archer C7)

Condo/Wave would offer typically symmetric 100 megabit on these lines, though the ability for more than a few customers on each "MoCA node" (for lack of a better term) to saturate them was much more limited

Another "fun" feature was that both the MoCA devices and switches they were connected to were run without any sort of VLAN'ing at all. If a customer accidentally plugged the MoCA link into the LAN port on their router, it would happily hand out DHCP leases to the entire building!

As I found out. The reason they don't use VLAN'ing is because their NOC staff are almost entirely customer service reps that were "upskilled" to handle NOC tasks (gaining a fixed $0.50 an hour bonus, hooray!). Wave's NOC handle roughly 90% of WaveG calls (I'd guess because Wave doesn't make very much money?)

One other fun anecdote:

WaveG service has a lot of users from overseas set up VPN's for their parents to watch Netflix on. Netflix's internal algorithms for the longest time would detect this behaviour and automatically flag Wave's entire IP ranges as a "proxy or VPN provider", knocking out roughly 500,000+ internet customers from using Netflix for several hours or even days. This would cause their phone support to effectively melt down, with the robotic queue time projecting at roughly 5-6 hours or more

Re: AT&T Fiber in the SF Bay Area is flipping bits

#296

Post by jl6 » Tue, Dec 08, 2020, 7:51 AM UTC

myself248 wrote: post #265

toast0 wrote: post #222

Earlier quoted context omitted.

> If there's anyone out there designing tech support procedures, you should add an "is this a 5% problem?" question to whatever checklist you give to support staff. When I was the engineer customer service escalated to, I was damn sure to thank them every time they escalated something. Even the one guy who escalated all the things I'd roll my eyes about in private. At least he was making sure the escalation path work…

Ex-phone-company here. (Is this the party to whom I am speaking?) I was in installation, but hung out with a lot of the ops crew, and they LOVED interesting problems. The trouble was getting such problems to the ops people in the first place. Good people, bad process. The most memorable one: Customer service had been getting calls all morning with a peculiar complaint: A customer's phone would ring, and when they ans…

Great story.

> including all the ephemera that would be forgotten when the call disconnects

Interesting to know there is information which is not logged. I’m guessing keeping this info, even for a day, would have helped isolate the issue?

How did the echo cancellers pass testing?

Re: AT&T Fiber in the SF Bay Area is flipping bits

#297

Post by robgolding » Tue, Dec 08, 2020, 7:59 AM UTC

iandanforth wrote: post #140

AaronFriel wrote: post #93

Trying to explain this issue to AT&T support is like trying to convince a doctor you're the only person on earth with a particular disease. Even explaining the issue is hard. It's not an outage, my internet isn't out , it's intermittently wrong . The phone support agents aren't prepared for this, and I can't find any way to escalate or speak to a network engineer. I feel like if I spoke to the right engineer, there'd…

Have you tried saying "shibboleet"?

My ISP actually supports this[0], though I haven’t had cause to use it yet - they are also very reliable!

[0]: https://www.aa.net.uk/broadband/why-choose-aaisp/

Re: AT&T Fiber in the SF Bay Area is flipping bits

#298

Post by jcims » Tue, Dec 08, 2020, 8:14 AM UTC

jl6 wrote: post #296

myself248 wrote: post #265

Earlier quoted context omitted.

Ex-phone-company here. (Is this the party to whom I am speaking?) I was in installation, but hung out with a lot of the ops crew, and they LOVED interesting problems. The trouble was getting such problems to the ops people in the first place. Good people, bad process. The most memorable one: Customer service had been getting calls all morning with a peculiar complaint: A customer's phone would ring, and when they ans…

Great story. > including all the ephemera that would be forgotten when the call disconnects Interesting to know there is information which is not logged. I’m guessing keeping this info, even for a day, would have helped isolate the issue? How did the echo cancellers pass testing?

Not OP but it sounds like the echo cancellers were fine, the interconnect to the switch was misconfigured. Rather than sending both channels of audio to opposite ends of the same call, one channel got directed to the next call.

The funny thing is that if everyone played along they could have had a mean game of telephone going.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#299

Post by havish » Tue, Dec 08, 2020, 8:17 AM UTC

myself248 wrote: post #265

toast0 wrote: post #222

Earlier quoted context omitted.

> If there's anyone out there designing tech support procedures, you should add an "is this a 5% problem?" question to whatever checklist you give to support staff. When I was the engineer customer service escalated to, I was damn sure to thank them every time they escalated something. Even the one guy who escalated all the things I'd roll my eyes about in private. At least he was making sure the escalation path work…

Ex-phone-company here. (Is this the party to whom I am speaking?) I was in installation, but hung out with a lot of the ops crew, and they LOVED interesting problems. The trouble was getting such problems to the ops people in the first place. Good people, bad process. The most memorable one: Customer service had been getting calls all morning with a peculiar complaint: A customer's phone would ring, and when they ans…

A very similar problem is happening currently in india with Jio. I wonder if anyone from there had seen this.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#300

Post by jcims » Tue, Dec 08, 2020, 8:19 AM UTC

myself248 wrote: post #265

toast0 wrote: post #222

Earlier quoted context omitted.

> If there's anyone out there designing tech support procedures, you should add an "is this a 5% problem?" question to whatever checklist you give to support staff. When I was the engineer customer service escalated to, I was damn sure to thank them every time they escalated something. Even the one guy who escalated all the things I'd roll my eyes about in private. At least he was making sure the escalation path work…

Ex-phone-company here. (Is this the party to whom I am speaking?) I was in installation, but hung out with a lot of the ops crew, and they LOVED interesting problems. The trouble was getting such problems to the ops people in the first place. Good people, bad process. The most memorable one: Customer service had been getting calls all morning with a peculiar complaint: A customer's phone would ring, and when they ans…

Amazing story! My step dad worked night shift at AT&T back in the 80’s and ran the 5ESS. He took my brother and I in for a tour one night. Thinking back on it now it was a lean crew for the equipment they were running. Rows and rows and rows of equipment. I don’t remember closed cabinets, mostly open frames moderately populated. I’ll never forget he showed us some magnetic core memory that was still mounted up on a frame in the switch room. Huuuge battery backup floor as well.

He loved all of that stuff, absolutely hated when everything went to computers. Quit and became a maintenance man at a nursing home, commercial laundry repair guy then finally retired this year in his late 70’s (due to Covid) after working maintenance at a local jail.