Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Level 3 Global Outage

puck.nether.net

Post reply on HN
Print view View original

261–270 of 393 posts

Re: Level 3 Global Outage

#261

Post by colde »

ystad wrote: post #226
emilstahl wrote: post #215

CenturyLink/Level3 on Twitter: "We are able to confirm that all services impacted by today’s IP outage have been restored. We understand how important these services are to our customers, and we sincerely apologize for the impact this outage caused." https://twitter.com/CenturyLink/status/1300089110858797063

I hope they provide a root cause analysis

Based on experience it will probably not public, or at least very limited.

But customers are likely to get one, at least if they request it.

Re: Level 3 Global Outage

#262

Post by phkahler »

beagle3 wrote: post #188
kitteh wrote: post #124

Massive reconvergence event in their network, causing edge router bgp sessions to bounce (due to cpu). Right now all their big peers are shutting down sessions with them to give level3s network the ability to reconverge. Prefixes announced to 3356 are frozen on their route reflectors and not getting withdrawn. Edit: if you are a Level3 customer shut your sessions down to them.

History doesn't repeat, but it rhymes .... There was a huge AT&T outage in 1990 that cut off most US long distance telephony (which was, at the time, mostly "everything not within the same area code"). It was a bug. It wasn't a reconvergence event, but it was a distant cousin: Something would cause a crash; exchanges would offload that something to other exchanges, causing them to crash -- but with enough time for th…

Contrary to what that link says, the software was not thoroughly tested. Normal testing was bypassed - per management request after a small code change.

This was covered in a book (perhaps Safeware, but maybe another one I dont recall) along with the Therac 25, the Ariane V, and several others. Unfortunately these lessons need to be relearned by each generation. See the 737-Max...

Re: Level 3 Global Outage

#263

Post by peterwwillis »

beagle3 wrote: post #188
kitteh wrote: post #124

Massive reconvergence event in their network, causing edge router bgp sessions to bounce (due to cpu). Right now all their big peers are shutting down sessions with them to give level3s network the ability to reconverge. Prefixes announced to 3356 are frozen on their route reflectors and not getting withdrawn. Edit: if you are a Level3 customer shut your sessions down to them.

History doesn't repeat, but it rhymes .... There was a huge AT&T outage in 1990 that cut off most US long distance telephony (which was, at the time, mostly "everything not within the same area code"). It was a bug. It wasn't a reconvergence event, but it was a distant cousin: Something would cause a crash; exchanges would offload that something to other exchanges, causing them to crash -- but with enough time for th…

I have spent hours and hours banging my head against Erlang distributed system bugs in production. I am absolutely mystified why anyone thought just using a particular programming language would prevent these scenarios. If it's Turing-complete, expect the unexpected.

Re: Level 3 Global Outage

#264

Post by kitteh »

lemiffe wrote: post #256

I had this earlier! A bunch of sites were down for me, I couldn't even connect to this site. The problem is I don't know where to find what was going on (tried looking up live DDOS-tracking websites, "is it down or is it just me" websites, etc. I couldn't find a single place talking about this. Is there a source where you can get instant information on Level3 / global DNS / major outages?

Ddos tracking sites are eye candy and garbage. Stop using them.

Outages and nanog lists are your best bet, short of being on the right IRC channels.

Re: Level 3 Global Outage

#265

Post by exikyut »

jbotz wrote: post #5
vld wrote: post #4

I'm having issues reaching IP addresses unrelated to Cloudflare. Based on some traceroutes, it seems AS174 (Cogent) and AS3356 (Level 3) are experiencing major outages.

Is there any one place that would be a good first place to go to check on outages like this? It would be really cool and useful to have an "public Internet health monitoring center"... this could be a foundation that gets some financing from industry that maintains a global internet health monitoring infrastructure and a central site at which all the major players announce outages. It would be pretty cheap and have a…

This is an excellent idea and simple but moderately expensive for anyone to set up.

Just have a site fetch resources from every single hosting provider everywhere. A 1x1 image would be enough, but 1K/100K/1M sized files might also be useful (they could also be crafted images)

The first step would be making the HTML page itself redundant. Strict round robin DNS might work well for that.

But yeah, moderately expensive - and... thinking about it... it'll honestly come in handy once every ten years? :/

Re: Level 3 Global Outage

#266

Post by MayeulC »

q3k wrote: post #252
MayeulC wrote: post #246

Earlier quoted context omitted.

> bgp.wtf, that's us I feel like you ought to be part of the ffdn database: https://db.ffdn.org/ Though the "French Data Networks Federation" is a French organization, their db tries to cover every independent, nonprofit ISP in the world :)

That couldn't work, unfortunately, per https://www.ffdn.org/en/charter-good-practices-and-common-co... : > All subscribers to Internet access provided by the provider must be members of the provider and must have the corresponding rights, including the right to vote, within reasonable time and procedure. Not all of our subscribers are members of our association. The association is primarily a hackerspace with hackers…

Ah, that's interesting, thank you for pointing this out to me, I didn't know about it. I take it that this isn't the first time you are asked about this, then?

Well, I on one hand I perfectly understand you not wanting to change your structure, especially if it works fine. On the other, can see a few ways around that restriction, and don't really see how having the ISP a separate association with its customers as members (maybe with their votes having less weight than hackerspace members) would have a downside (except if funds are primarilly collected for funding hackerspace activities?).

Re: Level 3 Global Outage

#267

Post by akritrime »

dboreham wrote: post #254
akritrime wrote: post #238

Earlier quoted context omitted.

What resources can I follow to start a non-profit ISP? I want to start one in my hometown for students who couldn't afford internet to join online classes.

Why not just raise money to pay for service from for-profit providers? Much more efficient use of donation funds.

Hmm, I actually didn't think about that at all. I guess I got too fascinated by this video[0] and wanted to apply something similar to our current scenario.

[0]: https://youtu.be/lEplzHraw3c

Re: Level 3 Global Outage

#268

Post by mmaunder »

aosaigh wrote: post #187

Has anyone any good resources for learning more about the "internet-level" infrastructure affected today and how global networks are connected?

Read Internet Routing Architectures by Sam Halabi. It’s almost 20 years old now but BGP hasn’t changed and the book is still called The Bible by routing architects.

Re: Level 3 Global Outage

#269

Post by bogomipz »

aosaigh wrote: post #187

Has anyone any good resources for learning more about the "internet-level" infrastructure affected today and how global networks are connected?

You want to learn about BGP in order to understand how routing on the internet works. The book "BGP" by Iljitsch van Beijnum is a great place to start. Don't be put off by the publication date, as almost everything in there is still relevant.[1]

Once you understand BGP and Autonomous Systems(AS), you can then understand peering as well as some of the politics that surround it.[2]

Then you can learn more about how specific networks are connected via public route servers and looking glass servers.[3][4][5]

Probably one of the best resource though still is to work for an ISP or other network provider for a stint.

[1] https://www.oreilly.com/library/view/bgp/9780596002541/

[2] http://drpeering.net/white-papers/Internet-Service-Providers...

[3] http://www.traceroute.org/#Looking%20Glass

[4] http://www.traceroute.org/#Route%20Servers

[5] http://www.routeviews.org/routeviews/

Re: Level 3 Global Outage

#270

Post by q3k »

MayeulC wrote: post #266
q3k wrote: post #252

Earlier quoted context omitted.

That couldn't work, unfortunately, per https://www.ffdn.org/en/charter-good-practices-and-common-co... : > All subscribers to Internet access provided by the provider must be members of the provider and must have the corresponding rights, including the right to vote, within reasonable time and procedure. Not all of our subscribers are members of our association. The association is primarily a hackerspace with hackers…

Ah, that's interesting, thank you for pointing this out to me, I didn't know about it. I take it that this isn't the first time you are asked about this, then? Well, I on one hand I perfectly understand you not wanting to change your structure, especially if it works fine. On the other, can see a few ways around that restriction, and don't really see how having the ISP a separate association with its customers as mem…

> I take it that this isn't the first time you are asked about this, then?

First time, but I read the rules carefully :).

> [I] don't really see how having the ISP a separate association with its customers as members [...] would have a downside [...]

Paperwork, in time and actual accounting fees. If/when we grow, this might happen - but for now it's just not worth the extra effort. We're not even breaking even on this, just using whatever extra income from customers to offset the costs of our own Internet infrastructure for the hackerspace. We don't even have enough customers to legally start a separate association with them as members, as far as I understand. I also don't think our customers would necessarily be interested in becoming members of an association, they just want good and cheap Internet access and/or mediocre and cheap colocation space.

Post reply on HN