Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Level 3 Global Outage

puck.nether.net

Post reply on HN
Print view View original

131–140 of 393 posts

Re: Level 3 Global Outage

#131

Post by lucb1e »

minxomat wrote: post #130
lucb1e wrote: post #122

Earlier quoted context omitted.

I see a lot of ads for NordVPN, but you should know they're not necessarily reliable. Just look for NordVPN on hacker news search: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... (see e.g. the second hit: https://news.ycombinator.com/item?id=21664692 covering up security issues, using your connection to proxy other people's traffic, a related company does data mining...). The only VPN that seemed to f…

I know. But they are required to unlock streaming services. I’m not using them for privacy or even normal traffic.

Alright, just making sure. Happy to hear you're an informed netizen :)

Re: Level 3 Global Outage

#133

Post by jsjohnst »

tpmx wrote: post #85

From the other (Cloudflare) thread (post: https://news.ycombinator.com/item?id=24322603 ), the outages list ( https://puck.nether.net/mailman/listinfo/outages ). https://puck.nether.net/pipermail/outages/2020-August/thread... Not a network engineer, but based on the comments there it looks like it's a BGP blackhole incident. Edit: removed details about the similarity to a 1997 incident based in input from commenters.

> Not a network engineer, but based on the comments there it looks like it's a BGP blackhole incident, possibly reminiscent of the https://en.wikipedia.org/wiki/AS_7007_incident in 1997.

As you aren’t a network engineer, I can understand making that leap based on the context, but no, this is nothing like the AS7007 event.

The “black hole” in this case is due to networks pulling their routes via AS3356 to try and avoid their outage, but when they do, CenturyLink is still announcing those routes and as such those networks blackhole.

Re: Level 3 Global Outage

#134

Post by [deleted] »

kitteh wrote: post #124

Massive reconvergence event in their network, causing edge router bgp sessions to bounce (due to cpu). Right now all their big peers are shutting down sessions with them to give level3s network the ability to reconverge. Prefixes announced to 3356 are frozen on their route reflectors and not getting withdrawn. Edit: if you are a Level3 customer shut your sessions down to them.

[deleted]

Re: Level 3 Global Outage

#135

Post by waheoo »

tpmx wrote: post #85

From the other (Cloudflare) thread (post: https://news.ycombinator.com/item?id=24322603 ), the outages list ( https://puck.nether.net/mailman/listinfo/outages ). https://puck.nether.net/pipermail/outages/2020-August/thread... Not a network engineer, but based on the comments there it looks like it's a BGP blackhole incident. Edit: removed details about the similarity to a 1997 incident based in input from commenters.

You're not a network engineer but it looks like a blackhole incident to you?

This is how people die in bushfires.

Re: Level 3 Global Outage

#136

Post by q3k »

regolithori wrote: post #89
iso1210 wrote: post #79

Looks like Centurylink/Level3 (as3356) might not be withdrawing routes after people close their peering?

What could cause this? I wonder what the technical problem is.

These are usually called 'BGP Zombies', and here's a good summary of their prevalence and usual causes: https://labs.ripe.net/Members/romain_fontugne/bgp-zombies

In this case however, it seems to be an L3/CL-specific bug.

Re: Level 3 Global Outage

#137

Post by tpmx »

waheoo wrote: post #135
tpmx wrote: post #85

From the other (Cloudflare) thread (post: https://news.ycombinator.com/item?id=24322603 ), the outages list ( https://puck.nether.net/mailman/listinfo/outages ). https://puck.nether.net/pipermail/outages/2020-August/thread... Not a network engineer, but based on the comments there it looks like it's a BGP blackhole incident. Edit: removed details about the similarity to a 1997 incident based in input from commenters.

You're not a network engineer but it looks like a blackhole incident to you? This is how people die in bushfires.

Heh, I knew I was setting myself up for that from networking people - i know the attitude. I was of course merely repeating the sentiment in that thread. What more disclaimers do you need to avoid displaying your superiority in networking? Sheesh.

Re: Level 3 Global Outage

#138

Post by tpmx »

jsjohnst wrote: post #133
tpmx wrote: post #85

From the other (Cloudflare) thread (post: https://news.ycombinator.com/item?id=24322603 ), the outages list ( https://puck.nether.net/mailman/listinfo/outages ). https://puck.nether.net/pipermail/outages/2020-August/thread... Not a network engineer, but based on the comments there it looks like it's a BGP blackhole incident. Edit: removed details about the similarity to a 1997 incident based in input from commenters.

> Not a network engineer, but based on the comments there it looks like it's a BGP blackhole incident, possibly reminiscent of the https://en.wikipedia.org/wiki/AS_7007_incident in 1997. As you aren’t a network engineer, I can understand making that leap based on the context, but no, this is nothing like the AS7007 event. The “black hole” in this case is due to networks pulling their routes via AS3356 to try and avoi…

So it's not a BGP blackhole incident then?

Re: Level 3 Global Outage

#139

Post by every »

abhishekjha wrote: post #93
Benjamin_Dobell wrote: post #88

This explains a lot. Initially thought my mobile phone Internet connectivity was flakey because I couldn't access HN here in Australia, whilst it's fine over wi-fi (wired Internet).

Its reverse for me. The broadband fails to connect to HN but my mobile ISP is able to reach it fine.

ycombinator.com pinged just fine but news.ycombinator.com dropped 100% packets. But all better now...

Re: Level 3 Global Outage

#140

Post by jsjohnst »

tpmx wrote: post #138
jsjohnst wrote: post #133

Earlier quoted context omitted.

> Not a network engineer, but based on the comments there it looks like it's a BGP blackhole incident, possibly reminiscent of the https://en.wikipedia.org/wiki/AS_7007_incident in 1997. As you aren’t a network engineer, I can understand making that leap based on the context, but no, this is nothing like the AS7007 event. The “black hole” in this case is due to networks pulling their routes via AS3356 to try and avoi…

So it's not a BGP blackhole incident then?

Not all BGP blackholes are the same. The AS7007 incident from over twenty years ago is an entirely different cause, and thus unrelated.
Post reply on HN