311–320 of 393 posts
Earlier quoted context omitted.
I believe these are mostly non public channels where backbone and network infrastructure engineers from different companies congregate to discuss outages like this.
also channels where hats of various type discuss advantages opportunities and challenges presented by such outages
Earlier quoted context omitted.
History doesn't repeat, but it rhymes .... There was a huge AT&T outage in 1990 that cut off most US long distance telephony (which was, at the time, mostly "everything not within the same area code"). It was a bug. It wasn't a reconvergence event, but it was a distant cousin: Something would cause a crash; exchanges would offload that something to other exchanges, causing them to crash -- but with enough time for th…
This reminds me of an incident on the early internet (perhaps ARPANET at that point) where a routing table got corrupted so it had a negative-length route which routers then propagated to each other, even after the original corrupt router was rebooted. As with AT&T, they had to reboot all the routers at once to get rid of the corruption. I can't remember where i read about this, but i recall the problem was called "T…
The incident is detailed in RFC 789:
Earlier quoted context omitted.
There are plenty of presentations out there. See nanog, ripe. The books are meh because they're not written by operators. They're more academic and dated. Plenty of clueful folks on the right IRC channels.
Second time in this thread you've mentioned the right IRC channels. Does one need to invoke some secret code to find out what they are? :-)
Earlier quoted context omitted.
also channels where hats of various type discuss advantages opportunities and challenges presented by such outages
Which channels
here are a couple of "for instance" breadcrumbs for you to start from:
https://github.com/saileshmittal/nodejs-irc-server
https://ubuntu.com/tutorials/irc-server
...>>> https://github.com/inspircd/inspircd/releases/tag/v3.7.0
That response time is atrocious. It wasn't that they needed to fix broken hardware, rather they needed to stop running hardware from actively sabotaging the global routing via the inherently insecure BGP protocol. That took 3-4 hours to happen.
As an example: Being in Sweden with an ISP that uses Telia Carrier for connectivity things started working around the time of https://twitter.com/TeliaCarrier/status/1300074378378518528
Earlier quoted context omitted.
> Also: BGP probably needs to redesigned from the ground up by software engineers with experience from designing systems that can remain working with hostile actors. This has been attempted a number of times, but this is a political problem, not a technical problem: there's no single agreed source of truth for routing policy. A lot of US Internet providers won't even sign up for ARIN IRR, or even move their legacy sp…
Build an industry coalition. Put pressure on those who don't join. Randomly throw away 1 out of 10000 packets from the providers that fail to get with the times. Increase that frequency according to some published time function.
Let's not forget that this is mainly a political problem and not a technical one. Would countries be willing to join a coalition with heavy influence from china for example? (or vice versa with the US).
Based on what I've seen: They essentially "shut down the Internet" for probably a quarter of the global population for about 3-4 hours. That response time is atrocious. It wasn't that they needed to fix broken hardware, rather they needed to stop running hardware from actively sabotaging the global routing via the inherently insecure BGP protocol. That took 3-4 hours to happen. As an example: Being in Sweden with an…
https://twitter.com/TeliaCarrier/status/1300074378378518528?...