Live data from Hacker News

Level 3 Global Outage

puck.nether.net

171–180 of 393 posts

Re: Level 3 Global Outage

#171
post #157

Earlier quoted context omitted.

I'd wager I'm not spreading disinforming because of said disclaimer and links to emails, but whatatever floats your boat.

Chris Adams made an exasperated comment[0] in the heat of the moment of dealing with an outage. You posting it here and positing it as a possibly valid information is the very definition of spreading misinformation and you did it for what gain? Maybe if multiple folks were saying it or at least agreeing with Chris, it might be different. Long story short, admit you’re wrong and just move on. [0] https://puck.nether.n…

[deleted]

Re: Level 3 Global Outage

#172

seems like the internet in 2020 has a diminished ability to route around damage

My opinion is that this, like many issues were seeing today, is largely an issue with ongoing consolidation trends. The less diversity of systems/solutions we have for a given problem or set if problems, the less chance we're protected from unknown unknowns that creep up. The more diversity you have in systems, the more likely you have some option that is hardened against unknown unknowns when they arrive and the quicker we can work around them.

Modern society is all about consolidating systems into a few efficient solutions typically dictated by market forces which I argue, don't concern themselves much with these sorts of problems. As a result, when we run into problems, we're left with fewer options to resort to and instead have to identify problems and develop new solutions on-the-fly. Consolidation leads to complacency and stagnation.

Sometimes this is reasonable (and even desirable) for certain non-critical systems, it just doesn't make financial sense to pour resources into system diversity for certain systems we could do without--find the one that works best/most efficiently and use it. If it breaks, it's not critical and the work around can wait.

On the other hand, if a system is critical, then I think it behooves us to continue looking at improvements of existing systems and alloting resources to investigating new approaches.

Re: Level 3 Global Outage

#173

So, that's why HN is unreachable from Belgium at the moment (right when I was trying to figure a dns cache problem in Firefox,of course). An ssh tunnel through OVH/gravelines is working so far. edit: Proximus. edit2: also, Orange Mobile

Was down from Latvia too, up now.

Re: Level 3 Global Outage

#174
Cloudflare status page: Update - Major transit providers are taking action to work around the network that is experiencing issues and affecting global traffic.

We are applying corrective action in our data centers as the situation changes in order to improve reachability Aug 30, 14:26 UTC

https://www.cloudflarestatus.com

Re: Level 3 Global Outage

#176
post #171

Earlier quoted context omitted.

Chris Adams made an exasperated comment[0] in the heat of the moment of dealing with an outage. You posting it here and positing it as a possibly valid information is the very definition of spreading misinformation and you did it for what gain? Maybe if multiple folks were saying it or at least agreeing with Chris, it might be different. Long story short, admit you’re wrong and just move on. [0] https://puck.nether.n…

[deleted]

> My current theory: The current Internet with the crappy BGP security model is the way it is because it was built by network admins turned amateur software developers. We need raise the level of software quality in the core parts of the net.

Further proving my point. Thanks and have a great day!

Re: Level 3 Global Outage

#177

seems like the internet in 2020 has a diminished ability to route around damage

BGP has always had this issue. It depends on trustworthy information being available. Any trusted source who starts lying (or just screws up) is going to cause routing problems.

Re: Level 3 Global Outage

#179
post #155
post #124

Massive reconvergence event in their network, causing edge router bgp sessions to bounce (due to cpu). Right now all their big peers are shutting down sessions with them to give level3s network the ability to reconverge. Prefixes announced to 3356 are frozen on their route reflectors and not getting withdrawn. Edit: if you are a Level3 customer shut your sessions down to them.

Most of level3s settlement free peers aka "tier 1s" have shutdown or depreffed their sessions with them. Example: https://mobile.twitter.com/TeliaCarrier/status/1300074378378...

Root cause identified. Folks are turning things back on now.
Post reply on HN