Live data from Hacker News

AT&T Fiber in the SF Bay Area is flipping bits

twitter.com

201–210 of 374 posts

Re: AT&T Fiber in the SF Bay Area is flipping bits

#201
post #59

Earlier quoted context omitted.

The parts of SV that have buried their power lines, probably haven't gotten a whole lot of new cable/fiber/DSL than whatever was there when they buried it. SV is the perfect place to not bury infrastructure. There's no weather to worry about, and mostly people aren't going to shoot down the fiber to try to claim scrap metals.

I live in one of the few cities in the Bay Area where everything is buried, and it's refreshing not to have stuff hanging all over the place and blocking the view. Now, granted, AT&T fiber is slow to come here, but it's hard to know if the fact that the infrastructure is buried is the main reason. They are in some areas of town but not others. You can bury conduits, by the way, and not cables or fiber directly. This…

Both options have trade-offs.

When I used to work in the telecom industry, burying conduit or cable in the ground was anywhere from 3x (bury in some dirt on the side of a county highway) to 20x (directional drilling in a heavily populated city where there's utilities all over the place) the cost per foot compared to hanging it aerially from a labor standpoint.

However, as you correctly point out, there may be restrictions on what you can hang on the poles and where, and oftentimes you'll find poles where it turns out it never should have had the number of attachments it did, but guess who gets to foot a large part of that bill if they want on?

But even then, I've seen absurd lengths gone to in the name of not digging. On Martha's Vineyard, I believe they wound up using a super-special Self-Supporting fiber that could be hung in or near the Power area of a Pole. Yes, that requires a far more trained/well paid worker than normal aerial work. Also, in that region, NESC 250C/D comes into play which makes it even more of a PITA. But it still was far cheaper than putting cable in the ground.

I wonder whether Teraspan or other Vertical Directed Conduit would be a good fit for the bay area (Saw-cut a minimal depth in the street, just lay in a special zip-up conduit for fiber or twisted pair.) If the weather doesn't tend towards large temperature shifts it works well.

Speaking of which, a couple drawbacks worth noting for buried conduit; You have to go out and do your markings, or pay someone to do them for you when a dig request is made, and you have to be ready to handle the repair when someone inevitably forgets to call or the markings are done incorrectly.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#202
post #8

We are AT&T in SF Bay Area, and our service has been flaking out for weeks. Inability to get DNS, machine claims invalid security certificates, download/upload drop below 1 MbS. All we could get is our modem reset.

I'm on Sonic.net fiber and they're a reseller of AT&T and I've experienced this exact scenario.

> they're a reseller of AT&T

Note: Only in certain areas. In SF they string their own fiber.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#203

From my professional experience of programming and debugging networking equipment, this could be a switch/router with a buffer with bad memory (stuck bit maybe). The better chips have CRC/Parity/ECC to cover such issues but there are always those magical choke points where the past CRC is tossed and the new one is generated that can leave a gaping hole. The tricky part is how often is this bad memory buffer used... I…

Why would a switch/router recalculate and rewrite the TCP checksum?

Re: AT&T Fiber in the SF Bay Area is flipping bits

#204

Sonic has their own fibre in San Francisco. Thank god for that. Life's too short for Comcast or AT&T.

Source? We have Sonic DSL and I see differences using the example.com test; and at installation time I was told I could only get fiber at my location (backside of Bernal) when AT&T put it in and they could lease it. That was in 2019. Do they own their own fiber in some areas?

https://thatdan.github.io/sonic_fiber/

Re: AT&T Fiber in the SF Bay Area is flipping bits

#205
post #193

Some new data.... I ran the same script on two different machines connected via the same AT&T provided modem. Machine A is NATted, B uses a public address. A always gets the error within 50-60 fetches. B doesn't (I ran the test a number of times for a few seconds each). I even ran them both concurrently (repeated the test on A since it keeps failing) but it was always A that got the error. I am wondering if this is m…

How are you bypassing the EAP authentication on your AT&T provided gateway? Are you using https://github.com/jaysoffian/eap_proxy ? My router is also getting a public IP address, I've set my AT&T gateway to bridge mode, but the AT&T gateway is still a monster-in-the-middle, a tumor I can't excise because it has to be between your router and the ONT or else the fiber turns off.

I recall hardly any of the details now but IIRC I talked to the AT&T support person and they changed some settings. Looking at the settings now, I see in "Home Networks->Subnets & DHCP" the [public subnet mode] is on. I haven't had to deal with any EAP authentication (but again I may be forgetting!).

And IP Passthrough is off. Basically the modem is routing between the public and private subnets.

BTW, I do have a static IP block from AT&T.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#206
post #76

You can contribute to our attempts to find the bad router card here: https://twitter.com/alexstamos/status/1336099461622157312 Almost certainly 12.242.117.22

I confirm that I am seeing these bit flips, and 12.242.117.22 is in my traceroute.

You people are amazing!

Re: AT&T Fiber in the SF Bay Area is flipping bits

#207
post #162

If you've got AT&T fiber via Sonic, give them a call and ask them to escalate the issue. The people on the phone know about the issue, take you seriously, and the more people complaining the more pressure they'll put on AT&T, I expect.

I have done this earlier today and it is exactly as you say. All they can do is file another ticket with AT&T, but multiple tickets coming to one ISP from another ISP mean more than calls or tweets to the AT&T customer support line, so here's hoping! Some people on Twitter report this problem going back all the way to November. Glad it's finally getting attention tonight.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#208

Not that I have any specific evidence this is the case, but it wouldnt surprise me to later learn this is an attack on encryption by the NSA. AT&T in San Francisco is ground 0 for some of their bulk collection. After The Times disclosed the Bush administration’s warrantless wiretapping program in December 2005, plaintiffs began trying to sue AT&T and the N.S.A. In a 2006 lawsuit, a retired AT&T technician named Mark…

Thank you for voicing this despite the downvotes. I don't think it's the most likely scenario, but it hurts to see how harshly people on HN reject speculation about other paths of possibility than the most obvious, especially considering AT&T's history.

Never forget: "NSA documents show that the relationship with AT&T has been considered unique and especially productive. One document described it as 'highly collaborative', while another lauded the company’s 'extreme willingness to help'."

Re: AT&T Fiber in the SF Bay Area is flipping bits

#209

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

> presumably because a corrupted packet in a TLS handshake is an error and the connection dies

Not just in the handshake, TLS moves these things called TLSPlaintext records (about 16kbytes each), not only in the handshake, but also for all the actual data - and they'll always have integrity protection to ensure bad guys can't change anything. TLS can't know the difference between a bad guy tampering with data and your crappy Internet mangling the data in transit, in either case the TLS protocol design says to "alert" bad_record_mac which your browser or similar software will probably treat as a failed connection, even if it happens mid-way through an HTTP transaction.

Because TLS guarantees integrity even if your fiber is a complete shit show, any TLSPlaintext records which do get from one end to the other are guaranteed to be as intended.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#210

Earlier quoted context omitted.

The problem is that the idiots the hire to do their "technical" support have zero skill (nor motivation to learn) to assess whether it's a 5% problem or not, and the majority of end-users aren't capable to answer that question either, nor that they are incentivized to answer truthfully. The solution could be a priority support tier where you pay upfront for an hour of a real network engineer's time (decently compensa…

I realize this was written from a position of frustration (which I share) at getting run around by customer support, but I'd reconsider the blanket characterization of tech support staff as "idiots": they're doing a high-throughput job following a playbook they're given with, as you identify, no incentive ---it's probably less about personal motivation than the expectations that are set for how they perform their job…

I cope with level 1 support by remembering we have a common goal: to stop talking to each other as quickly as possible. The tech just wants to close the case and I want to talk to someone else who can actually help me.
Post reply on HN