Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AT&T Fiber in the SF Bay Area is flipping bits

twitter.com

Post reply on HN
Print view View original

181–190 of 374 posts

Re: AT&T Fiber in the SF Bay Area is flipping bits

#181

Post by astrange »

jedberg wrote: post #26
benlivengood wrote: post #12

It's ironic that for the most part Silicon Valley only has terrible ISP infrastructure. Cellular service isn't all that great either.

NIMBYism at its finest. Cupertino did not allow cell phone towers for a very long time. The only one was an ATT tower on the top of Infinite Loop, right on the Sunnyvale border. The people who would call their provider about bad cell coverage in their house are the same people that would go to city hall and demand that no cell towers be built in the city.

It’s amusing seeing Cupertino city council transcripts about this because the people show up claiming 5G gives them cancer and the city council desperately tries to get them to use better excuses so they can approve denying it.

Makes a change from the city council members usual practice of denying Vallco permits and claiming Apple employees are hiring prostitutes and molesting high school students. (I did not make this up.)

Re: AT&T Fiber in the SF Bay Area is flipping bits

#182

Post by brigade »

ProfessorLayton wrote: post #47

I had AT&T via Sonic's FTTN in an effort to avoid Comcast/AT&T directly, especially since Sonic does not have data caps despite running on AT&T's network and actually respects their customer's privacy. Unfortunately my experience was so terrible and unreliable that I decided to give up and finally sign up with Comcast, and just have to mind that 1.2TB data cap. The Bay Area of all places should not have such terrible…

It's shitty, but just FYI: you can pay Comcast an additional $30/mo for unlimited data. They hide the option, but it was ultimately the least bad option to handle my WFH traffic after Comcast re-instated the caps.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#184

Post by [deleted] »

cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

[deleted]

Re: AT&T Fiber in the SF Bay Area is flipping bits

#185

Post by p1necone »

noncoml wrote: post #133
cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

Much scarier when you actually see it with your own eyes: $ diff example-* 4c4 Example Domain --- > Example Domain 11c11 background-color: #f0f0f2; 23c23 box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02); 41c41 This domain is for use in illustrative examples in documents. You may usm this --- > This domain is for use in illustrative examples in documents. You may use this

I was under the impression there was enough layers of checksums once you get to the level of UDP/TCP that these kind of single bit flips should be detected and dropped before you get to read them. What's going on here? Is networking gear not calculating checksums like it should?

Re: AT&T Fiber in the SF Bay Area is flipping bits

#186

Post by PieUser »

maerF0x0 wrote: post #159

Not that I have any specific evidence this is the case, but it wouldnt surprise me to later learn this is an attack on encryption by the NSA. AT&T in San Francisco is ground 0 for some of their bulk collection. After The Times disclosed the Bush administration’s warrantless wiretapping program in December 2005, plaintiffs began trying to sue AT&T and the N.S.A. In a 2006 lawsuit, a retired AT&T technician named Mark…

LOL

Re: AT&T Fiber in the SF Bay Area is flipping bits

#187

Post by monkeybutton »

sneak wrote: post #178

Requisite link to the excellent paper by Artem Dinaburg about Bitsquatting, the practice of passively exploiting single bit flips (usually in ram, but on the wire works too): https://media.blackhat.com/bh-us-11/Dinaburg/BH_US_11_Dinabu...

I guess I'm one of the lucky 10,000 today; thanks for posting this!

Re: AT&T Fiber in the SF Bay Area is flipping bits

#188

Post by city41 »

proverbialbunny wrote: post #65
ProfessorLayton wrote: post #47

I had AT&T via Sonic's FTTN in an effort to avoid Comcast/AT&T directly, especially since Sonic does not have data caps despite running on AT&T's network and actually respects their customer's privacy. Unfortunately my experience was so terrible and unreliable that I decided to give up and finally sign up with Comcast, and just have to mind that 1.2TB data cap. The Bay Area of all places should not have such terrible…

It's because AT&T does not follow net neutrality. It has fast lanes and slow lanes, which cause a lot of problems for work VPNs even when down the street. This TLS handshake bit flip is just yet another issue AT&T has, including the 1.1.1.1 DNS issues, and everything else. I'm on Sonic.net fiber over AT&T too. To get the equivalent fiber from Comcast out here it's $270 a month. However, if you're lucky enough Sonic.n…

Huh, I had Sonic fiber over AT&T and loved it. In May I moved out of the bay area and now have Comcast cable. I'd do pretty much anything to get Sonic's level of quality out here.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#189

Post by mitchs »

beervirus wrote: post #137
cpach wrote: post #132

Earlier quoted context omitted.

How do you mean? How would it know if a bit in the HTTP payload got flipped?

Checksums. TCP is supposed to provide reliability against data corruption, resend bad packets, etc.

Unfortunately TCP checksums are hot garbage given switch ASIC design. They are a 16 bit one's complement sum over a packet. If you get two bit flips in the same offset % 16, you can pass a checksum.

The problem is routers slow down the high speed serial signals from fiber to by splitting the bits over a large number of slower speed signals internally. Often those wider busses are a multiple of 16 bits. For example, one ASIC I know of moves things around in 204 byte chunks. (Might have been 208, been a while.) Anyway, the problem is that if there is a defect in one of those parallel elements it will always flip bits in the same offset position mod 204 bytes, which is the same position mod 16 bits. If the hardware is degraded enough, it can end up flipping two bits in the same position, and that has a fairly good chance of passing the checksum.

Ethernet has proper CRCs on packets, which is a lot less vulnerable to shenanigans like this, but unfortunately those can end up being checked on the way in, discarded, and then re-generated on the way out of a router. If anything is corrupted in the middle of the switch ASIC, nothing notices and it passes along. I once helped troubleshoot an issue in our network where a BGP packet was corrupted in this way. The flipped bits ended up causing a more specific route to be generated, and we had the world weirdest BGP route hijack within the bounds of our own data center.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#190

Post by _whiteCaps_ »

iandanforth wrote: post #140
AaronFriel wrote: post #93

Trying to explain this issue to AT&T support is like trying to convince a doctor you're the only person on earth with a particular disease. Even explaining the issue is hard. It's not an outage, my internet isn't out , it's intermittently wrong . The phone support agents aren't prepared for this, and I can't find any way to escalate or speak to a network engineer. I feel like if I spoke to the right engineer, there'd…

Have you tried saying "shibboleet"?

I think you're being unfairly downvoted by people who haven't seen this xkcd:

https://xkcd.com/806/

Post reply on HN