Live data from Hacker News

Level 3 Global Outage

puck.nether.net

361–370 of 393 posts

Re: Level 3 Global Outage

#364
post #328

Observation: You're not really adjusting your language to the target audience. (Edited away potentially hurtful language.) Learn from Feynman. Explain things using concepts the target audience can be expected to understand. Real mastery of a concept is when you can explain it using simple terms to any other reasonably intelligent person.

We detached this subthread from https://news.ycombinator.com/item?id=24325352.

Re: Level 3 Global Outage

#365

Earlier quoted context omitted.

Contrary to what that link says, the software was not thoroughly tested. Normal testing was bypassed - per management request after a small code change. This was covered in a book (perhaps Safeware, but maybe another one I dont recall) along with the Therac 25, the Ariane V, and several others. Unfortunately these lessons need to be relearned by each generation. See the 737-Max...

> Normal testing was bypassed - per management request after a small code change. That lesson will really never be learned. This happens on a daily basis all over the planet with people who have not been bitten - yet.

That's why the most reliable way to instil this lesson is to instil it into our tools. Automate as much testing as possible, so that bypassing the tests becomes more work than running them.

Re: Level 3 Global Outage

#366
Can anyone help me understand why I can't access HN from my iPhone, but I can from my computer? both are on the same network. I'm getting "Safari cannot open the page because the server cannot be found", and many apps won't work at all either.

Re: Level 3 Global Outage

#367

Can anyone help me understand why I can't access HN from my iPhone, but I can from my computer? both are on the same network. I'm getting "Safari cannot open the page because the server cannot be found", and many apps won't work at all either.

One might be using IPv6 and the other v4. Or you might have different DNS settings.

Re: Level 3 Global Outage

#368
post #367

Can anyone help me understand why I can't access HN from my iPhone, but I can from my computer? both are on the same network. I'm getting "Safari cannot open the page because the server cannot be found", and many apps won't work at all either.

One might be using IPv6 and the other v4. Or you might have different DNS settings.

[deleted]

Re: Level 3 Global Outage

#370
post #124

Massive reconvergence event in their network, causing edge router bgp sessions to bounce (due to cpu). Right now all their big peers are shutting down sessions with them to give level3s network the ability to reconverge. Prefixes announced to 3356 are frozen on their route reflectors and not getting withdrawn. Edit: if you are a Level3 customer shut your sessions down to them.

History doesn't repeat, but it rhymes .... There was a huge AT&T outage in 1990 that cut off most US long distance telephony (which was, at the time, mostly "everything not within the same area code"). It was a bug. It wasn't a reconvergence event, but it was a distant cousin: Something would cause a crash; exchanges would offload that something to other exchanges, causing them to crash -- but with enough time for th…

A thread discussing that event:

https://news.ycombinator.com/item?id=24323412

Post reply on HN