In a situation like this, what are the best "status" sites to be watching?
HN is not the worst place, honestly.
HN was down for me, unfortunately. (Connecting from Japan, so most CDN-based website load fine since it isn't route via Europe)
Level 3 Global Outage
281–290 of 393 posts
Re: Level 3 Global Outage
#282A service I run on Digital Ocean was affected by this early this morning. Looks like it was mitigated by DO - so I'm very grateful for that. Although, the service I run is time sensitive so failures like this are pretty unfortunate for me. Where would I get started with building in redundancy against these sort of outages?
Re: Level 3 Global Outage
#283I had this earlier! A bunch of sites were down for me, I couldn't even connect to this site. The problem is I don't know where to find what was going on (tried looking up live DDOS-tracking websites, "is it down or is it just me" websites, etc. I couldn't find a single place talking about this. Is there a source where you can get instant information on Level3 / global DNS / major outages?
I agree! I'm definitely an amateur when it comes to networking stuff. At the time, the _only_ issue I had was with all of my Digital Ocean droplets. It was confusing because I was able to get to them through my LTE connection and not able to through my home ISP. I opened a ticket with DO worried that it was my ISP blocking IP addresses suddenly. It turned out to be this outage, but it was very specific. Traceroute ga…
I had a similar issue with my droplets as well. I thought I messed up something and then suddenly it worked again.
Re: Level 3 Global Outage
#284> "Root Cause: An offending flowspec announcement prevented BGP from establishing correctly, impacting client services."
--
That doesn't really explain the "stuck" routes in their RRs... maybe it'll make sense once we've gotten some more details...
Re: Level 3 Global Outage
#285Massive reconvergence event in their network, causing edge router bgp sessions to bounce (due to cpu). Right now all their big peers are shutting down sessions with them to give level3s network the ability to reconverge. Prefixes announced to 3356 are frozen on their route reflectors and not getting withdrawn. Edit: if you are a Level3 customer shut your sessions down to them.
History doesn't repeat, but it rhymes .... There was a huge AT&T outage in 1990 that cut off most US long distance telephony (which was, at the time, mostly "everything not within the same area code"). It was a bug. It wasn't a reconvergence event, but it was a distant cousin: Something would cause a crash; exchanges would offload that something to other exchanges, causing them to crash -- but with enough time for th…
I think we used to call that a poison pill message (still bring it up routinely when we talk about load balancing and why infinite retries are a very, very bad idea).
Re: Level 3 Global Outage
#286Earlier quoted context omitted.
Ddos tracking sites are eye candy and garbage. Stop using them. Outages and nanog lists are your best bet, short of being on the right IRC channels.
What are the right IRC channels?
I believe these are mostly non public channels where backbone and network infrastructure engineers from different companies congregate to discuss outages like this.
Re: Level 3 Global Outage
#287Has anyone any good resources for learning more about the "internet-level" infrastructure affected today and how global networks are connected?
US and Israel looking at China's strategy at BGP-level in 2018 :
Re: Level 3 Global Outage
#288Earlier quoted context omitted.
History doesn't repeat, but it rhymes .... There was a huge AT&T outage in 1990 that cut off most US long distance telephony (which was, at the time, mostly "everything not within the same area code"). It was a bug. It wasn't a reconvergence event, but it was a distant cousin: Something would cause a crash; exchanges would offload that something to other exchanges, causing them to crash -- but with enough time for th…
Contrary to what that link says, the software was not thoroughly tested. Normal testing was bypassed - per management request after a small code change. This was covered in a book (perhaps Safeware, but maybe another one I dont recall) along with the Therac 25, the Ariane V, and several others. Unfortunately these lessons need to be relearned by each generation. See the 737-Max...
Probably not the book you are thinking off, since it’s just about the AT&T incident, but “The Day the Phones Stopped Ringing” by Leonard Lee is a detailed description of the event.
It’s been many years since I read it, but I recall it being a very interesting read.
Re: Level 3 Global Outage
#289Re: Level 3 Global Outage
#290Earlier quoted context omitted.
Unfortunately, this infrastructure is at an uncanny intersection of technology, business and politics. To learn the technical aspect of it, you can follow any network engineering certification materials or resources that delve into dynamic routing protocols, notably BGP. Inter-ISP networking is nothing but setting up BGP sessions and filters at the technical level. Why you set these up, and under what conditions is a…
What resources can I follow to start a non-profit ISP? I want to start one in my hometown for students who couldn't afford internet to join online classes.