You can contribute to our attempts to find the bad router card here: https://twitter.com/alexstamos/status/1336099461622157312 Almost certainly 12.242.117.22
Confirming this IP is in my traceroute to example.com as well
AT&T Fiber in the SF Bay Area is flipping bits
211–220 of 374 posts
Re: AT&T Fiber in the SF Bay Area is flipping bits
#212Earlier quoted context omitted.
And conversely its always surprising and disarming when you call a company and actually get through to a knowledgable employee. I was so surprised to hear “thats a firmware bug we know about and there is no update yet” about my router issue that I forgot to be mad at the company for not caring my router is broken.
FWIW, this has the hallmarks of an interaction within the context of an abusive interpersonal relationship.
[deleted]
Re: AT&T Fiber in the SF Bay Area is flipping bits
#213Earlier quoted context omitted.
All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…
And conversely its always surprising and disarming when you call a company and actually get through to a knowledgable employee. I was so surprised to hear “thats a firmware bug we know about and there is no update yet” about my router issue that I forgot to be mad at the company for not caring my router is broken.
I had a problem with a PowerMac G3 back in the day and I somehow managed to escalate up to tier 3 which is to say an Apple HW engineer. He was brusque bordering on rude, but he immediately recognized it was a problem with an undocumented jumper setting on the motherboard and solved my problem inside of two minutes. It definitely increased my customer satisfaction.
Re: AT&T Fiber in the SF Bay Area is flipping bits
#214Earlier quoted context omitted.
And some routing hardware have been known to ignore this--meaning they'll forward the packet data along and re-calculate the checksum.
Hardware that forwards packets usually forwards IP packets, it doesn't care about TCP checksums and doesn't recalculate them. TCP stack in your OS is the one that does that. It has to be a special kind of hardware that does much deeper packet inspection (DPI) to recalculate TCP checksums, usually used for spying, throttling, censorship, injecting ads, injecting exploits, etc., but not merely routing/forwarding packet…
Switches (L2 devices) recalculate Ethernet CRCs (L2 error detection codes), and routers tend to calculate TCP/UDP checksums (L3 error detection codes) and everything below. I've seen exactly this issue with switches that have bad RAM before, and I assume that they just have a router with bad RAM (and without ECC RAM, which looks embarrassing).
Re: AT&T Fiber in the SF Bay Area is flipping bits
#215Earlier quoted context omitted.
I realize this was written from a position of frustration (which I share) at getting run around by customer support, but I'd reconsider the blanket characterization of tech support staff as "idiots": they're doing a high-throughput job following a playbook they're given with, as you identify, no incentive ---it's probably less about personal motivation than the expectations that are set for how they perform their job…
I cope with level 1 support by remembering we have a common goal: to stop talking to each other as quickly as possible. The tech just wants to close the case and I want to talk to someone else who can actually help me.
That’s a great example of finding how you’re aligned and then using it to get a mutually beneficial outcome.
Re: AT&T Fiber in the SF Bay Area is flipping bits
#216Earlier quoted context omitted.
Have you tried saying "shibboleet"?
I think you're being unfairly downvoted by people who haven't seen this xkcd: https://xkcd.com/806/
The real life version of “shibboleet” is your Certified Partner ID number and a serial number with a valid support contract.
When I worked for a VAR I could upload logs to Cisco and get experimental patches back. Call up HP, tell them I want an RMA, and they’d just do it. Night and day compared to what consumers get.
Re: AT&T Fiber in the SF Bay Area is flipping bits
#217I had AT&T via Sonic's FTTN in an effort to avoid Comcast/AT&T directly, especially since Sonic does not have data caps despite running on AT&T's network and actually respects their customer's privacy. Unfortunately my experience was so terrible and unreliable that I decided to give up and finally sign up with Comcast, and just have to mind that 1.2TB data cap. The Bay Area of all places should not have such terrible…
It's shitty, but just FYI: you can pay Comcast an additional $30/mo for unlimited data. They hide the option, but it was ultimately the least bad option to handle my WFH traffic after Comcast re-instated the caps.
You can get Comcast Business at your residence. The overall bandwidth is lower, but you're priority traffic, and there are no caps. Plus, static IPv4 and IPv6 are available.
Re: AT&T Fiber in the SF Bay Area is flipping bits
#218Earlier quoted context omitted.
When I first moved to Seattle there was a great local ISP called CondoInternet that mainly specialized in high-density downtown buildings. I was once having some packet loss issues and called their support line. I assume the company was really small at the time, because the guy who answered the phone was clearly a network engineer who knew the system inside and out. I read him a couple of traceroutes over the phone a…
And when Wave bought them, Wave put incompetent people in charge. I moved to a Seattle suburb and had "Wave G" (post-Wave CondoInternet). I had throughput issues, first on my apartment. That got resolved via Reddit. Then the backbone is slow from time to time which persisted. Sometimes I get a Gigabit, other times I only get 10-20 Mbps. What did Wave say? Let's bring a tech out. I told them "it's a backbone issue" an…
I don't have much to add here, except to say that this really matches my experience with Wave G.
Wave G's support team also doesn't even realize that the CondoInternet network they acquired provides IPv6, and when asked about IPv6-related issues they just say "oh we don't have IPv6 yet" which is nonsense. I really miss the local CondoInternet support people. They were amazing.
Re: AT&T Fiber in the SF Bay Area is flipping bits
#219https://en.wikipedia.org/wiki/Room_641A gone wrong?
Re: AT&T Fiber in the SF Bay Area is flipping bits
#220If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS ha…
All of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there des…
Way back in the day I worked on equipment that straddled telco circuits. T1s, E1s, DS3s, OC whatever. Companies paying big money for those circuits.
Anyway I was told on more than one occasion by different telcos that the standard operating procedure for many techs was to take the call, do nothing, and call back 20 minutes later and ask if it looked better because ... often enough it did.