Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AT&T Fiber in the SF Bay Area is flipping bits

twitter.com

Post reply on HN
Print view View original

131–140 of 374 posts

Re: AT&T Fiber in the SF Bay Area is flipping bits

#131

Post by jamilbk »

The script linked in the gist (https://gist.github.com/bmastenbrook/14c0e22fc02b95d4a48f82d...) is reporting diffs for me on Sonic fiber as well, which is leased from AT&T in much of the Bay Area. So seems to be a network layer or below issue.

Interestingly, the incorrect ASCII chars are always 8 chars higher or lower than their counterpart, indicating that bit 3 is consistently being flipped:

  flip: f|0b1100110
  orig: n|0b1101110
  ---
  flip: i|0b1101001
  orig: a|0b1100001
  ---
  flip: 6|0b110110
  orig: >|0b111110
  ---
  flip: (|0b101000
  orig:  |0b100000
  ---
  flip: p|0b1110000
  orig: x|0b1111000
  ---
  flip: m|0b1101101
  orig: e|0b1100101
Edit: Can confirm connecting to VPN (ProtonVPN in this case) solves the issue.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#132

Post by cpach »

beervirus wrote: post #123
cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

Why would a TCP connection allow flipped bits to make it through?

How do you mean? How would it know if a bit in the HTTP payload got flipped?

Re: AT&T Fiber in the SF Bay Area is flipping bits

#133

Post by noncoml »

cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

Much scarier when you actually see it with your own eyes:

    $ diff example-*
    4c4
    Example Domain
    ---
    >     Example Domain
    11c11
             background-color: #f0f0f2;
    23c23
             box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);
    41c41
    This domain is for use in illustrative examples in documents. You may usm this
    ---
    >     
This domain is for use in illustrative examples in documents. You may use this

Re: AT&T Fiber in the SF Bay Area is flipping bits

#134

Post by sidewndr46 »

cpach wrote: post #132
beervirus wrote: post #123

Earlier quoted context omitted.

Why would a TCP connection allow flipped bits to make it through?

How do you mean? How would it know if a bit in the HTTP payload got flipped?

Each TCP packet has a checksum but it does not catch 100% of all possible bit flips

Re: AT&T Fiber in the SF Bay Area is flipping bits

#135

Post by Bluecobra »

dudul wrote: post #120
warent wrote: post #107

Earlier quoted context omitted.

The obnoxious thing to me is the hubris that must be behind this. Either they considered it and decided they would never encounter a system error like this and refused to implement an escalation route, or they never even considered it. Or perhaps even worse than that, maybe they considered it, decided it was possible, but just don't care because of their insane borderline monopoly. I don't understand how internet com…

Wait so you have your main cable plugged into their modem, their router plugged into their modem, and then your own router plugged into their router?

Funnily enough this is how my AT&T fiber is set up as well. They force you to use their router, and you can’t directly connect your router to the ONT. The problem is that they use a device certificate + EAP. There’s work arounds but it’s a pain.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#136

Post by baggy_trough »

noncoml wrote: post #133
cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

Much scarier when you actually see it with your own eyes: $ diff example-* 4c4 Example Domain --- > Example Domain 11c11 background-color: #f0f0f2; 23c23 box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02); 41c41 This domain is for use in illustrative examples in documents. You may usm this --- > This domain is for use in illustrative examples in documents. You may use this

I get the same. Pretty amazing that the only effect from such garbled data that I noticed were some annoying hangs and bad Twitter links!

Re: AT&T Fiber in the SF Bay Area is flipping bits

#137

Post by beervirus »

cpach wrote: post #132
beervirus wrote: post #123

Earlier quoted context omitted.

Why would a TCP connection allow flipped bits to make it through?

How do you mean? How would it know if a bit in the HTTP payload got flipped?

Checksums. TCP is supposed to provide reliability against data corruption, resend bad packets, etc.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#138

Post by runawaybottle »

p1necone wrote: post #105
cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…

Anecdote from my past:

I used to play counter strike/Starcraft in my middle school years. I pretty much figured out I had consistent packet loss with a simple ping test. I was on the phone with Time Warner every other day for months. They kept sending the regular technicians, at one point ripping out all of my cable wires in the house to see if it fixed the problem. Nothing worked, I kept calling, at this point I had the direct number to level 3 support. They saw the packet loss too. Finally, after two or three months they send out a Chief Engineer. Guy says I’ll look at the ‘box’ on one of the cable poles down the block. He confirmed something was wrong at that source for the whole area. Then it finally got fixed.

Took forever dealing with level 1 support, and lots of karenesque ‘can I talk to your supervisor please’, but that’s literally what it took.

So yeah, if you want stuff like this fixed, stay professional, never ever curse, consistently ask to speak to the supervisor, keep records, and keep calling.

Small shout out to the old http://www.dslreports.com/ for being a great support community during the early days of broadband for consumer activism in terms of making sure you got legit good broadband.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#139

Post by garmaine »

sidewndr46 wrote: post #134
cpach wrote: post #132

Earlier quoted context omitted.

How do you mean? How would it know if a bit in the HTTP payload got flipped?

Each TCP packet has a checksum but it does not catch 100% of all possible bit flips

And some routing hardware have been known to ignore this--meaning they'll forward the packet data along and re-calculate the checksum.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#140

Post by iandanforth »

AaronFriel wrote: post #93

Trying to explain this issue to AT&T support is like trying to convince a doctor you're the only person on earth with a particular disease. Even explaining the issue is hard. It's not an outage, my internet isn't out , it's intermittently wrong . The phone support agents aren't prepared for this, and I can't find any way to escalate or speak to a network engineer. I feel like if I spoke to the right engineer, there'd…

Have you tried saying "shibboleet"?
Post reply on HN