Live data from Hacker News

AT&T Fiber in the SF Bay Area is flipping bits

twitter.com

51–60 of 374 posts

Re: AT&T Fiber in the SF Bay Area is flipping bits

#51
post #19

Earlier quoted context omitted.

Sonic's fiber service uses AT&T fiber in Alameda as well. Just another data point. I have that service. It's been working just fine, though. I haven't noticed any symptoms of flipped bits or anything similar to what others are reporting here.

Oh man, I didn't even realize they were available outside of SF with fiber.

Ya, my Sonic fiber service is in San Mateo.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#52

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

> or maybe someone from AT&T is on here that can get it looked at...

They are on Facebook and Linkedin, you have seem them.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#54

From my professional experience of programming and debugging networking equipment, this could be a switch/router with a buffer with bad memory (stuck bit maybe). The better chips have CRC/Parity/ECC to cover such issues but there are always those magical choke points where the past CRC is tossed and the new one is generated that can leave a gaping hole. The tricky part is how often is this bad memory buffer used... I…

”I would use traceroute to find a common bad point for everyone“

How do you mean? I use traceroute from time to time but I’m not sure how it would apply in a case like this. Feel free to elaborate :)

Re: AT&T Fiber in the SF Bay Area is flipping bits

#55

It's ironic that for the most part Silicon Valley only has terrible ISP infrastructure. Cellular service isn't all that great either.

Palo Alto has its own municipal fiber program: https://www.cityofpaloalto.org/gov/depts/utl/business/progra... Wonder if someone should take them up on it.

It’s only for commercial use. As much as I’d love to be able to drop a node in to peer at paix for my house ...

Re: AT&T Fiber in the SF Bay Area is flipping bits

#56

I once had a combination of "dueling" VPN clients on a Windows PC that caused random bit flips. It was awful. SSL/TLS sessions randomly failing. SSH sessions closing randomly. Downloaded files failing integrity checks. It's an interesting juxtaposition of how tolerant TCP/IP is of random bit flips, yet how intolerant application layer protocols are.

Surely bit flips are invisible to application layer protocols?

Re: AT&T Fiber in the SF Bay Area is flipping bits

#57

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

I ran the script and it is finding a difference after a while. I have not noticed anything wrong with the network recently, but that of course does not mean there has not been a problem.

Edit: Actually I take that back, I have seen one of the issues where sometimes web sites will not load at random, and a reload fixes the issue.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#58

Earlier quoted context omitted.

The parts of SV that haven't buried their power lines (not too pretty to begin with) have gotten significantly uglier since all of the new cable/fiber/DSL infrastructure has gone in. There are in many cases multiple of these things on loads of poles: https://www.dslreports.com/forum/r31593060-Equip-What-is-thi...

Here in Germany we bury everything . I don't even want to imagine how long it takes to rebuild all of that when some drunkard crashes his car into one of these poles or a lightning strike hits.

I grew up in the Florida Keys where the water table is very close to the surface so it's impossible to bury anything. When I was a young teenager, we would lose electricity all the time from tree branches brushing up against the power lines. They finally solved the problem by putting in power lines on concrete poles that are a few stories up! No more squirrels and tree branches disputing power!

Re: AT&T Fiber in the SF Bay Area is flipping bits

#59
post #16

Earlier quoted context omitted.

The ISPs have tried, but they get pushback from local residents whenever they try to install the necessary network boxes at intervals down the street. They had to go out of their way to hide cellular towers as streetlamps to get the 5G rollout to happen; I remember getting two or three notices for this (each from different telcos, so the one street corner has three streetlamps and two traffic lights on one corner). I…

The parts of SV that haven't buried their power lines (not too pretty to begin with) have gotten significantly uglier since all of the new cable/fiber/DSL infrastructure has gone in. There are in many cases multiple of these things on loads of poles: https://www.dslreports.com/forum/r31593060-Equip-What-is-thi...

The parts of SV that have buried their power lines, probably haven't gotten a whole lot of new cable/fiber/DSL than whatever was there when they buried it.

SV is the perfect place to not bury infrastructure. There's no weather to worry about, and mostly people aren't going to shoot down the fiber to try to claim scrap metals.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#60

From my professional experience of programming and debugging networking equipment, this could be a switch/router with a buffer with bad memory (stuck bit maybe). The better chips have CRC/Parity/ECC to cover such issues but there are always those magical choke points where the past CRC is tossed and the new one is generated that can leave a gaping hole. The tricky part is how often is this bad memory buffer used... I…

It seems like it, but it's a widespread issue across the SF/Bay Area right now, maybe wider. I've been having it for weeks and exploring it as well. I've even gone as far as ripping the certs off of the router to double check.
Post reply on HN