Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Level 3 Global Outage

puck.nether.net

Post reply on HN
Print view View original

81–90 of 393 posts

Re: Level 3 Global Outage

#81

Post by omnibrain »

minxomat wrote: post #75
lucb1e wrote: post #72

Earlier quoted context omitted.

HN works fine from Germany with Telefonica (O2) and also from the Netherlands with XS4ALL. Edit: Somewhere between 14:00 and 14:46Z it also went down from O2; XS4ALL still works, and O2 can reach XS4ALL.

No luck on T-Mobile

Yes, I had to switch to my Vodafone eSIM for data to connect to Hacker News.

Re: Level 3 Global Outage

#82

Post by eric_khun »

wiremine wrote: post #63

In a situation like this, what are the best "status" sites to be watching?

Currently working on a project[1] to monitor all the 3rd party stack you use for your services. Hit me up if you want, access I'll give free access for a year+ to some folks to get feedbacks.

[1] https://monitory.io

Re: Level 3 Global Outage

#83

Post by iso1210 »

johnchristopher wrote: post #77

So, that's why HN is unreachable from Belgium at the moment (right when I was trying to figure a dns cache problem in Firefox,of course). An ssh tunnel through OVH/gravelines is working so far. edit: Proximus. edit2: also, Orange Mobile

HN working for me from the UK on BT, but traceroute showing lots of different bouncing around and a lot of different hops in the US

  7  166-49-209-132.gia.bt.net (166.49.209.132)  9.877 ms  8.929 ms
    166-49-209-131.gia.bt.net (166.49.209.131)  8.975 ms
  8  166-49-209-131.gia.bt.net (166.49.209.131)  8.645 ms  10.323 ms  10.434 ms
  9  be12497.ccr41.par01.atlas.cogentco.com (154.54.56.130)  95.018 ms
    be3487.ccr41.lon13.atlas.cogentco.com (154.54.60.5)  7.627 ms
    be12497.ccr41.par01.atlas.cogentco.com (154.54.56.130)  102.570 ms
  10  be3627.ccr41.jfk02.atlas.cogentco.com (66.28.4.197)  89.867 ms
    be12497.ccr41.par01.atlas.cogentco.com (154.54.56.130)  101.469 ms  101.655 ms
  11  be2806.ccr41.dca01.atlas.cogentco.com (154.54.40.106)  103.990 ms  93.885 ms
    be3627.ccr41.jfk02.atlas.cogentco.com (66.28.4.197)  97.525 ms
  12  be2112.ccr41.atl01.atlas.cogentco.com (154.54.7.158)  106.027 ms
    be2806.ccr41.dca01.atlas.cogentco.com (154.54.40.106)  98.149 ms  97.866 ms
  13  be2687.ccr41.iah01.atlas.cogentco.com (154.54.28.70)  120.558 ms  122.330 ms  120.071 ms
  14  be2687.ccr41.iah01.atlas.cogentco.com (154.54.28.70)  123.662 ms
    be2927.ccr21.elp01.atlas.cogentco.com (154.54.29.222)  128.351 ms
    be2687.ccr41.iah01.atlas.cogentco.com (154.54.28.70)  120.746 ms
 15  be2929.ccr31.phx01.atlas.cogentco.com (154.54.42.65)  145.939 ms  137.652 ms
    be2927.ccr21.elp01.atlas.cogentco.com (154.54.29.222)  128.043 ms
  16  be2930.ccr32.phx01.atlas.cogentco.com (154.54.42.77)  150.015 ms
    be2940.rcr51.san01.atlas.cogentco.com (154.54.6.121)  152.793 ms  152.720 ms
  17  be2941.rcr52.san01.atlas.cogentco.com (154.54.41.33)  152.881 ms
    te0-0-2-0.rcr11.san03.atlas.cogentco.com (154.54.82.66)  153.452 ms
    be2941.rcr52.san01.atlas.cogentco.com (154.54.41.33)  152.054 ms
  18  te0-0-2-0.rcr12.san03.atlas.cogentco.com (154.54.82.70)  162.835 ms
    te0-0-2-0.nr11.b006590-1.san03.atlas.cogentco.com (154.24.18.190)  146.643 ms
    te0-0-2-0.rcr12.san03.atlas.cogentco.com (154.54.82.70)  153.714 ms
  19  te0-0-2-0.nr11.b006590-1.san03.atlas.cogentco.com (154.24.18.190)  151.212 ms  145.735 ms
    38.96.10.250 (38.96.10.250)  147.092 ms
  20  38.96.10.250 (38.96.10.250)  149.413 ms * *

Re: Level 3 Global Outage

#84

Post by josephb »

iso1210 wrote: post #79

Looks like Centurylink/Level3 (as3356) might not be withdrawing routes after people close their peering?

That's what various networks have reported.

It kind of makes it hard to route around an upstream, if they keep announcing your routes even when there isn't a path to you!

Re: Level 3 Global Outage

#85

Post by tpmx »

From the other (Cloudflare) thread (post: https://news.ycombinator.com/item?id=24322603), the outages list (https://puck.nether.net/mailman/listinfo/outages).

https://puck.nether.net/pipermail/outages/2020-August/thread...

Not a network engineer, but based on the comments there it looks like it's a BGP blackhole incident.

Edit: removed details about the similarity to a 1997 incident based in input from commenters.

Re: Level 3 Global Outage

#87

Post by toomuchtodo »

lucb1e wrote: post #78
dkdk8283 wrote: post #69

Earlier quoted context omitted.

Nanog is also pretty helpful for this specific type of issue

You mean nanog.org? I don't see a stats page linked in their menu.

It’s a mailing list for network operations/engineering folks. The emails are the status updates. You’ll have to look to each network’s own site if you want connectivity, peering, and IXP red/green/ up/down status.
Post reply on HN