Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AT&T Fiber in the SF Bay Area is flipping bits

twitter.com

Post reply on HN
Print view View original

321–330 of 374 posts

Re: AT&T Fiber in the SF Bay Area is flipping bits

#321

Post by mschuster91 »

orlp wrote: post #294
pakl wrote: post #271

Earlier quoted context omitted.

That’s why you need to sign up for CSA Pre™. Get preauthorized for instant escalation on customer support calls. All you have to do is answer a form with questions like: Do you know how to plug in a computer? Do you know where the power switches are on your devices? etc. CSA Pre™ is valid for 5 years; you can initiate the renewal process to do 6 months before expiry.

At this point we unironically need shibboleet. https://xkcd.com/806/

Many "phone robot" systems can be overridden by mashing on the keyboard, shouting or swearing - these will get redirected to a live human ASAP.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#322

Post by comboy »

cosmie wrote: post #304
comboy wrote: post #253

Earlier quoted context omitted.

Which is then a problem when you try to explain them that yes, you are sure the issue is with their service and not your setup. But what's your reason to be going such lengths instead of just plugging UDM into their router? Unless it was done for the fun of it which is fine and understandable.

> But what's your reason to be going such lengths instead of just plugging UDM into their router? While you can do this and things will generally work, AT&T restricts all of their residential gateways from operating in a true passthrough/bridge mode to another router. So you end with double NAT and all the joys that entails (such as [1]). There are also a number of other issues that have been associated with operatin…

Thanks for such a detailed reply.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#323

Post by myself248 »

jcims wrote: post #313
myself248 wrote: post #303

Earlier quoted context omitted.

They passed testing because they had each been individually crossconnected to a test trunk, and test calls force-routed over that trunk. Then to place them in service, the crossconnects were reconfigured to place them at their normal location in the system. The testing was to prove the voice path of each DSP card, and that those cards were wired into the crossconnect properly. All that was true, the failure happened…

Were the cross-connected circuits channelized or individual voice calls (ds0? I can’t remember from my wan days) or something else?

As I recall, the cross-connects were done at the DS1 level, and an individual card handled 24 calls. These are hazy, hazy memories now; this took place in 2004-ish.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#324

Post by superjan »

tialaramex wrote: post #209
cameldrv wrote: post #43

If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…

> presumably because a corrupted packet in a TLS handshake is an error and the connection dies Not just in the handshake, TLS moves these things called TLSPlaintext records (about 16kbytes each), not only in the handshake, but also for all the actual data - and they'll always have integrity protection to ensure bad guys can't change anything. TLS can't know the difference between a bad guy tampering with data and you…

Thanks for sharing this! I have long suspected that TLS does this. It is great: for many applications preventing bitflips in transit is arguably more important than privacy. Is there an authorative source where this is documented?

Re: AT&T Fiber in the SF Bay Area is flipping bits

#325

Post by taurath »

14u2c wrote: post #312
m463 wrote: post #223

Earlier quoted context omitted.

I had a ginormous AT&T router/modem (pace 5268ac) with a set of static ip addresses and a few times, AT&T just stopped routing traffic to it. It had happened before and then magically fixed itself a few days later. One time I had a week of outage with AT&T basically said the problem was on my side. They could ping the modem, and then punted. I had several truck rolls. The techs were really nice guys, but were basical…

That is absurd that AT&T requires the use of a rented gateway for U-verse. I've never had an issue with a another provider refusing to support off the self hardware before, including with a DSL provider, multiple cable companies, and FiOS (Ethernet on ONT).

They like customer data

Re: AT&T Fiber in the SF Bay Area is flipping bits

#326

Post by jcims »

myself248 wrote: post #323
jcims wrote: post #313

Earlier quoted context omitted.

Were the cross-connected circuits channelized or individual voice calls (ds0? I can’t remember from my wan days) or something else?

As I recall, the cross-connects were done at the DS1 level, and an individual card handled 24 calls. These are hazy, hazy memories now; this took place in 2004-ish.

Nice!! Thanks for the walk down memory lane, this was cool.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#327

Post by reader_1000 »

AaronFriel wrote: post #93

Trying to explain this issue to AT&T support is like trying to convince a doctor you're the only person on earth with a particular disease. Even explaining the issue is hard. It's not an outage, my internet isn't out , it's intermittently wrong . The phone support agents aren't prepared for this, and I can't find any way to escalate or speak to a network engineer. I feel like if I spoke to the right engineer, there'd…

Ten years ago, while using ADSL, for some reason captcha images were not loading. I opened a case to ISP support and they called me back. The support guy did not believe it. He said "this is not an analog network, this is digital, you either have it or not". I said I know what the analog and digital mean and also I know this is a digital network since I am a computer engineer and I checked everything so this was a issue with connection. After a couple hours later, he called me back and said that problem was caused by modem and a driver update would fix it and it did. These were good old days, however, when we can reach someone on the phone and talk about problem. Nowadays everything is either automated response or some random person whose whole job to tell you that he/she cannot do anything about the situation.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#328

Post by grumple »

cbhl wrote: post #16
benlivengood wrote: post #12

It's ironic that for the most part Silicon Valley only has terrible ISP infrastructure. Cellular service isn't all that great either.

The ISPs have tried, but they get pushback from local residents whenever they try to install the necessary network boxes at intervals down the street. They had to go out of their way to hide cellular towers as streetlamps to get the 5G rollout to happen; I remember getting two or three notices for this (each from different telcos, so the one street corner has three streetlamps and two traffic lights on one corner). I…

SV residents really do complain about everything, huh? Try living in an east coast city, our aesthetic is a mess.

Re: AT&T Fiber in the SF Bay Area is flipping bits

#329

Post by segfaultbuserr »

secalex wrote: post #41
jedberg wrote: post #5

Oh man, I thought it was just my crappy old router causing the problems! I've just been too lazy to call them to replace it. I'm going to tweet at them right now. Edit: Reading the tweet thread, classic interaction: ATT says "please click this to check your connection", guy replies back and says "I think we know more about networks that you do, please get a network admin in here".

What would you know about production networking, Jeremy? I'm in the same boat. If the dozens of IT pros who are complaining about this can't get AT&T to swap out a single router card, what hope do most folks have?

Fortunately, this malfunction occurred in the SF Bay Area, getting AT&T to fix its network is only a matter of time, sooner or later, the correct person in the Valley would be alerted.

If it happened to the rest of us at somewhere else, we are probably out of luck...

Re: AT&T Fiber in the SF Bay Area is flipping bits

#330

Post by LeonM »

orlp wrote: post #294
pakl wrote: post #271

Earlier quoted context omitted.

That’s why you need to sign up for CSA Pre™. Get preauthorized for instant escalation on customer support calls. All you have to do is answer a form with questions like: Do you know how to plug in a computer? Do you know where the power switches are on your devices? etc. CSA Pre™ is valid for 5 years; you can initiate the renewal process to do 6 months before expiry.

At this point we unironically need shibboleet. https://xkcd.com/806/

Story time!

I've had this exact experience, except that it wasn't a dream.

Back in 2010 I had a weird issue where my cable connection would sometimes completely block the connectoin right after the DHCP response (we had dynamic IPs back then). This would go on for a couple of hours, until the IP lease expired, then my connection would come back. Luckily, I was running an OpenBSD box as my router which allowed me to diagnose the problem. But it was also impossible to explain to the servicedesk employees.

One evening it happened again, and I called the servicedesk, totally prepared to do the 'yes I have turned it off and on again' dance. But to my surprise the employee that I got on the phone was very knowledgable and even said that it was very cool that I had an OpenBSD box as a router. He very quickly diagnosed that someone in my neighbourhood was 'hammering' the DHCP service by not releasing his lease (a common trick to keep your IP address somewhat static). This caused a double IP on the subnet, and the L2 switch to block traffic to my port.

He asked me "do you know how to spoof your MAC with an OpenBSD box?". Then I knew this guy was legit. He instructed me to replace the last 2 bytes of the MAC with AB:BA (named after the music group). They had a separate DHCP pool for MAC addresses in that range. If they ever saw an ABBA mac address on their network, they knew it was someone who had connectivity issues before.

The problem was immediately solved, and I had a rock-solid internet connection for years, with a static IP!

I ended up chatting about networking and OpenBSD a bit, before I (as humble as I could) told the guy I was a bit flabbergasted that someone as knowledgable as him was working on the servicedesk.

It turned out, he was the chief of network operations at the ISP (the biggest ISP in my country). He was just manning the phone while some of his colleagues from the servicedesk were having diner.

Sometimes miracles do happen.

Post reply on HN