Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Level 3 Global Outage

puck.nether.net

Post reply on HN
Print view View original

331–340 of 393 posts

Re: Level 3 Global Outage

#331

Post by srosmd »

tpmx wrote: post #328

Observation: You're not really adjusting your language to the target audience. (Edited away potentially hurtful language.) Learn from Feynman. Explain things using concepts the target audience can be expected to understand. Real mastery of a concept is when you can explain it using simple terms to any other reasonably intelligent person.

What was the intent of posting this? This is an article on a global network outage - some folks want the technical nitty-gritty and others don't. You seem adversarial or pretentious when you unexpectedly post things like this even if your intentions are well-meaning.

Re: Level 3 Global Outage

#332

Post by tpmx »

srosmd wrote: post #331
tpmx wrote: post #328

Observation: You're not really adjusting your language to the target audience. (Edited away potentially hurtful language.) Learn from Feynman. Explain things using concepts the target audience can be expected to understand. Real mastery of a concept is when you can explain it using simple terms to any other reasonably intelligent person.

What was the intent of posting this? This is an article on a global network outage - some folks want the technical nitty-gritty and others don't. You seem adversarial or pretentious when you unexpectedly post things like this even if your intentions are well-meaning.

I agree it was needlessly adversarial (sorry about that!) - but it got the desired effect - an excellent explanation of the concept with lots of relevant background information (thanks, kitteh, upvoted). I think this helps the discussion a lot since a lot more people would be able to join. Less gatekeeping.

Re: Level 3 Global Outage

#333

Post by tpmx »

kitteh wrote: post #329
tpmx wrote: post #328

Observation: You're not really adjusting your language to the target audience. (Edited away potentially hurtful language.) Learn from Feynman. Explain things using concepts the target audience can be expected to understand. Real mastery of a concept is when you can explain it using simple terms to any other reasonably intelligent person.

BGP is a routing protocol that is mostly used for propagating routing/reachability information that also includes additional data that can be used (communities as tags, etc). A few years ago folks wanted to bake in additional functionality. For example, packet filters (aka ACLs) normally are deployed to router configuration files using each operators own tooling. To deploy this against hundreds or thousands of router…

Thanks! Excellent backgrounder.

Re: Level 3 Global Outage

#334

Post by matsur »

swinglock wrote: post #320
tpmx wrote: post #318

Based on what I've seen: They essentially "shut down the Internet" for probably a quarter of the global population for about 3-4 hours. That response time is atrocious. It wasn't that they needed to fix broken hardware, rather they needed to stop running hardware from actively sabotaging the global routing via the inherently insecure BGP protocol. That took 3-4 hours to happen. As an example: Being in Sweden with an…

Seems they didn't even get around to doing so, rather asking other carriers to stop peering with them. https://twitter.com/TeliaCarrier/status/1300074378378518528?...

CenturyLink requested depeering to give them some breathing room and stop the bleeding. Hug ops.

Re: Level 3 Global Outage

#335

Post by rmdashrfstar »

kitteh wrote: post #329
tpmx wrote: post #328

Observation: You're not really adjusting your language to the target audience. (Edited away potentially hurtful language.) Learn from Feynman. Explain things using concepts the target audience can be expected to understand. Real mastery of a concept is when you can explain it using simple terms to any other reasonably intelligent person.

BGP is a routing protocol that is mostly used for propagating routing/reachability information that also includes additional data that can be used (communities as tags, etc). A few years ago folks wanted to bake in additional functionality. For example, packet filters (aka ACLs) normally are deployed to router configuration files using each operators own tooling. To deploy this against hundreds or thousands of router…

Thanks for taking the time to write all of this out!

Re: Level 3 Global Outage

#336

Post by tinco »

phkahler wrote: post #262
beagle3 wrote: post #188

Earlier quoted context omitted.

History doesn't repeat, but it rhymes .... There was a huge AT&T outage in 1990 that cut off most US long distance telephony (which was, at the time, mostly "everything not within the same area code"). It was a bug. It wasn't a reconvergence event, but it was a distant cousin: Something would cause a crash; exchanges would offload that something to other exchanges, causing them to crash -- but with enough time for th…

Contrary to what that link says, the software was not thoroughly tested. Normal testing was bypassed - per management request after a small code change. This was covered in a book (perhaps Safeware, but maybe another one I dont recall) along with the Therac 25, the Ariane V, and several others. Unfortunately these lessons need to be relearned by each generation. See the 737-Max...

For some reason in my university almost every CS class would start with an anecdote about the Therac 25, Ariane V, and/or a couple others as a motivation on why we the class existed. It was sort of a meme.

The lessons are definitely still taught, I don't know if they're actually learned of course.. And who knows who actually taught the 737-Max software devs, I don't suppose they're fresh out of uni.

Re: Level 3 Global Outage

#337

Post by quickthrower2 »

mikro2nd wrote: post #97

Had to laugh: "I'm seeing complaints from all over the planet on Twitter" The one site I can't see is Twitter. (Not a heart-wrenching loss, mind you...)

I could not get on HN as a logged in person (logged out was OK) during this. I wondered how big the cloudflare thread would be if people could get on to comment on it :-)

Re: Level 3 Global Outage

#338

Post by quickthrower2 »

jlgaddis wrote: post #284

> "Root Cause: An offending flowspec announcement prevented BGP from establishing correctly, impacting client services." -- That doesn't really explain the "stuck" routes in their RRs... maybe it'll make sense once we've gotten some more details...

This might be a silly question but is there such a thing as CI/CD for this sort of thing that may have caught the problem?
Post reply on HN