Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Level 3 Global Outage

puck.nether.net

Post reply on HN
Print view View original

341–350 of 393 posts

Re: Level 3 Global Outage

#341

Post by mikelward »

hinkley wrote: post #285
beagle3 wrote: post #188

Earlier quoted context omitted.

History doesn't repeat, but it rhymes .... There was a huge AT&T outage in 1990 that cut off most US long distance telephony (which was, at the time, mostly "everything not within the same area code"). It was a bug. It wasn't a reconvergence event, but it was a distant cousin: Something would cause a crash; exchanges would offload that something to other exchanges, causing them to crash -- but with enough time for th…

I think we used to call that a poison pill message (still bring it up routinely when we talk about load balancing and why infinite retries are a very, very bad idea).

Some queue processing systems I've seen have infinite retries.

At least they have exponential backoff I guess.

Re: Level 3 Global Outage

#342

Post by [deleted] »

kitteh wrote: post #329
tpmx wrote: post #328

Observation: You're not really adjusting your language to the target audience. (Edited away potentially hurtful language.) Learn from Feynman. Explain things using concepts the target audience can be expected to understand. Real mastery of a concept is when you can explain it using simple terms to any other reasonably intelligent person.

BGP is a routing protocol that is mostly used for propagating routing/reachability information that also includes additional data that can be used (communities as tags, etc). A few years ago folks wanted to bake in additional functionality. For example, packet filters (aka ACLs) normally are deployed to router configuration files using each operators own tooling. To deploy this against hundreds or thousands of router…

[deleted]

Re: Level 3 Global Outage

#343

Post by jrockway »

lemiffe wrote: post #256

I had this earlier! A bunch of sites were down for me, I couldn't even connect to this site. The problem is I don't know where to find what was going on (tried looking up live DDOS-tracking websites, "is it down or is it just me" websites, etc. I couldn't find a single place talking about this. Is there a source where you can get instant information on Level3 / global DNS / major outages?

I found places talking about this earlier. A friend of mine who has CenturyLink as their ISP complained to me that Twitch and Reddit weren't working. But they worked for me, so I suspected a CDN issue. I did some digging to figure out what CDNs they had in common. I expected Twitch to be on CloudFront, but their CDN doesn't serve CloudFront headers; instead they are "Via: 1.1 varnish". Reddit is exactly the same. I did some googling and found out that they both apparently used Fastly, at least to some extent. Fastly has a status page and it was talking about "widespread disruption".

So I guess my takeaway from this is that if the Internet seems to be down, usually the CDN providers notice. I don't know if either of the sites actually still use Fastly (I kind of forgot they existed), but I did end up reading about the Internet being broken at some scale larger than "your friend's cable modem is broken", so that was helpful.

It would be nice if we had a map of popular sites and which CDN they use, so we can collect a sampling of what's up and what's down and figure out which CDN is broken. Though in this case, it wasn't really the CDN's fault. Just collateral damage.

Re: Level 3 Global Outage

#344

Post by ngold »

eastdakota wrote: post #339

Analysis of what we saw at Cloudflare, how our systems automatically mitigated the worst of the impact to our customers, and some speculation on what may have gone wrong: https://blog.cloudflare.com/analysis-of-todays-centurylink-l...

Great write up. It is embarrassing that most of America has no competition in the market.

>To use the old Internet as a “superhighway” analogy, that’s like only having a single offramp to a town. If the offramp is blocked, then there’s no way to reach the town. This was exacerbated in some cases because CenturyLink/Level(3)’s network was not honoring route withdrawals and continued to advertise routes to networks like Cloudflare’s even after they’d been withdrawn. In the case of customers whose only connectivity to the Internet is via CenturyLink/Level(3), or if CenturyLink/Level(3) continued to announce bad routes after they'd been withdrawn, there was no way for us to reach their applications and they continued to see 522 errors until CenturyLink/Level(3) resolved their issue around 14:30 UTC. The same was a problem on the other (“eyeball”) side of the network. Individuals need to have an onramp onto the Internet’s superhighway. An onramp to the Internet is essentially what your ISP provides. CenturyLink is one of the largest ISPs in the United. Because this outage appeared to take all of the CenturyLink/Level(3) network offline, individuals who are CenturyLink customers would not have been able to reach Cloudflare or any other Internet provider until the issue was resolved. Globally, we saw a 3.5% drop in global traffic during the outage, nearly all of which was due to a nearly complete outage of CenturyLink’s ISP service across the United States.

Re: Level 3 Global Outage

#345

Post by ZWoz »

aosaigh wrote: post #187

Has anyone any good resources for learning more about the "internet-level" infrastructure affected today and how global networks are connected?

DrPeering is good material: http://drpeering.net/tools/HTML_IPP/ipptoc.html

Geoff Huston paper "Interconnection, Peering, and Settlements" is older, but still interesting and several ways relevant.

I suggest "Where Wizards Stay Up Late: The Origins Of The Internet" - generic and talks about Internet history, but mentions several common misconseptions.

Re: Level 3 Global Outage

#346

Post by [deleted] »

matsur wrote: post #334
swinglock wrote: post #320

Earlier quoted context omitted.

Seems they didn't even get around to doing so, rather asking other carriers to stop peering with them. https://twitter.com/TeliaCarrier/status/1300074378378518528?...

CenturyLink requested depeering to give them some breathing room and stop the bleeding. Hug ops.

[deleted]

Re: Level 3 Global Outage

#347

Post by jacquesm »

phkahler wrote: post #262
beagle3 wrote: post #188

Earlier quoted context omitted.

History doesn't repeat, but it rhymes .... There was a huge AT&T outage in 1990 that cut off most US long distance telephony (which was, at the time, mostly "everything not within the same area code"). It was a bug. It wasn't a reconvergence event, but it was a distant cousin: Something would cause a crash; exchanges would offload that something to other exchanges, causing them to crash -- but with enough time for th…

Contrary to what that link says, the software was not thoroughly tested. Normal testing was bypassed - per management request after a small code change. This was covered in a book (perhaps Safeware, but maybe another one I dont recall) along with the Therac 25, the Ariane V, and several others. Unfortunately these lessons need to be relearned by each generation. See the 737-Max...

> Normal testing was bypassed - per management request after a small code change.

That lesson will really never be learned. This happens on a daily basis all over the planet with people who have not been bitten - yet.

Re: Level 3 Global Outage

#348

Post by tpmx »

matsur wrote: post #334
swinglock wrote: post #320

Earlier quoted context omitted.

Seems they didn't even get around to doing so, rather asking other carriers to stop peering with them. https://twitter.com/TeliaCarrier/status/1300074378378518528?...

CenturyLink requested depeering to give them some breathing room and stop the bleeding. Hug ops.

That is a fantastic euphemism. Personally I'm disappointed Telia didn't de-peer two hours earlier, after diagnosing the issue for 30 minutes, since that whole lack of functioning routning to very large parts of the internet forced me to use VPN in north america to access many web services, including HN.

I realize I'm going to get insanely downvoted by the elite internetworking crowd again but I think this needs to be said.

From an outsider's POV: There seems to be a very strange and almost incestual relationship between the networking companies. Or maybe it's just their hangaround supporters? I dunno.

Re: Level 3 Global Outage

#349

Post by hinkley »

mikelward wrote: post #341
hinkley wrote: post #285

Earlier quoted context omitted.

I think we used to call that a poison pill message (still bring it up routinely when we talk about load balancing and why infinite retries are a very, very bad idea).

Some queue processing systems I've seen have infinite retries. At least they have exponential backoff I guess.

But your queue will grow and grow and the fraction of time you spend servicing old messages grows and grows.

Not a terribly big fan of these queueing systems. People always seem to bung things up in ways they are not quite equipped to fix (in the “you are not smart enough to debug the code you wrote” sense).

Last time I had to help someone with such a situation, we discovered that the duplicate processing problem had existed for >3 months prior to the crisis event, and had been consuming 10% of the system capacity, which was just low enough that nobody noticed.

Re: Level 3 Global Outage

#350

Post by edoceo »

jacquesm wrote: post #347
phkahler wrote: post #262

Earlier quoted context omitted.

Contrary to what that link says, the software was not thoroughly tested. Normal testing was bypassed - per management request after a small code change. This was covered in a book (perhaps Safeware, but maybe another one I dont recall) along with the Therac 25, the Ariane V, and several others. Unfortunately these lessons need to be relearned by each generation. See the 737-Max...

> Normal testing was bypassed - per management request after a small code change. That lesson will really never be learned. This happens on a daily basis all over the planet with people who have not been bitten - yet.

Managers who have been bitten still make this choice
Post reply on HN