Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AT&T Fiber in the SF Bay Area is flipping bits

twitter.com

Post reply on HN
Print view View original

21–30 of 374 posts

Re: AT&T Fiber in the SF Bay Area is flipping bits

#21

Post by floatingatoll »

cbhl wrote: post #14

When AT&T first did their Fiber rollout in SF, one of the things I remember was they charged you $10 extra if you didn't want them to MITM all your connections to insert JavaScript pointing to their own ads. They rolled this back when folks complained, but I wonder if the relevant infrastructure is still sitting around and mangling packets. I would probably expect this to be some network card or cable or connector is…

Yes, as of six months ago I had to look up how to opt out of MITMing for a coworker on AT&T Fiber, so that they could use the Internet again when the remote-activated MITM feature inside the modem broke.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#23

Post by floatingatoll »

jmount wrote: post #8

We are AT&T in SF Bay Area, and our service has been flaking out for weeks. Inability to get DNS, machine claims invalid security certificates, download/upload drop below 1 MbS. All we could get is our modem reset.

What’s the hostname on the invalid security certificates?

Re: AT&T Fiber in the SF Bay Area is flipping bits

#24

Post by ribosometronome »

Bud wrote: post #19
gricardo99 wrote: post #4

Earlier quoted context omitted.

That may not make any difference, depending on your locality. Unfortunately AT&T infrastructure is still relied on by others such as Sonic, which leases lines from the likes of AT&T fiber, at least in my locality.

Sonic's fiber service uses AT&T fiber in Alameda as well. Just another data point. I have that service. It's been working just fine, though. I haven't noticed any symptoms of flipped bits or anything similar to what others are reporting here.

Oh man, I didn't even realize they were available outside of SF with fiber.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#26

Post by jedberg »

benlivengood wrote: post #12

It's ironic that for the most part Silicon Valley only has terrible ISP infrastructure. Cellular service isn't all that great either.

NIMBYism at its finest. Cupertino did not allow cell phone towers for a very long time. The only one was an ATT tower on the top of Infinite Loop, right on the Sunnyvale border.

The people who would call their provider about bad cell coverage in their house are the same people that would go to city hall and demand that no cell towers be built in the city.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#27

Post by gopalv »

fnord77 wrote: post #18
benlivengood wrote: post #12

It's ironic that for the most part Silicon Valley only has terrible ISP infrastructure. Cellular service isn't all that great either.

is the cellular deadspot on 101 just north of redwood city still there?

Are you thinking about the area just around the SFO landing zone?

I thought that was less of a "won't" and more of a "can't" between the lack of taller building towers and the radio interference zone for the airport means the directional antennas skip a slice there.

This will probably get fixed if 5G becomes a proper thing and there's a lot of micro-cells along the "bay area's biggest parking lot".

Re: AT&T Fiber in the SF Bay Area is flipping bits

#28

Post by StillBored »

cbhl wrote: post #14

When AT&T first did their Fiber rollout in SF, one of the things I remember was they charged you $10 extra if you didn't want them to MITM all your connections to insert JavaScript pointing to their own ads. They rolled this back when folks complained, but I wonder if the relevant infrastructure is still sitting around and mangling packets. I would probably expect this to be some network card or cable or connector is…

The tweet indicates it is at a specific bit position. That isn't symptomatic of a bad cable/sfp/etc. Analog problems like that tend to be more random. Random bit flipping in a fixed bit position is symptomatic of bad ram somewhere, or a router asic gone bad, or various software/config issues.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#29

Post by francoisLabonte »

From my professional experience of programming and debugging networking equipment, this could be a switch/router with a buffer with bad memory (stuck bit maybe). The better chips have CRC/Parity/ECC to cover such issues but there are always those magical choke points where the past CRC is tossed and the new one is generated that can leave a gaping hole. The tricky part is how often is this bad memory buffer used...

I would use traceroute to find a common bad point for everyone. It is also possible that the networking point where the problem occurs is invisible to traceroute as it could be part of a provider network probably MPLS but at least the common ends of the tunnel would be visible.

The fact that it is a a specific interval indicates a stuck bit in memory.

Some good previous public stories about such incidents https://www.verizondigitalmedia.com/blog/being-good-stewards... https://twitter.com/cperciva/status/1309568337408454658

Post reply on HN