Level 3 Global Outage • Hacker News Board

Re: Level 3 Global Outage

#391

Post by parliament32 » Tue, Sep 01, 2020, 5:36 PM UTC

kitteh wrote: post #325

dz0ny wrote: post #321

Summary: On August 30, 2020 10:04 GMT, CenturyLink identified an issue to be affecting users across multiple markets. The IP Network Operations Center (NOC) was engaged, and initial research identified that an offending flowspec announcement prevented Border Gateway Protocol (BGP) from establishing across multiple elements throughout the CenturyLink Network. The IP NOC deployed a global configuration change to block…

Flowspec strikes again. Its a super useful tool if you want to blast out an ACL across your network in seconds (using BGP) but it has a number of sharp edges. Several networks, including Cloudflare have learned what it can do. I've seen a few networks basically blackhole traffic or even lock themselves out of routers due to a poorly made Flowspec rules or a bug in the implementation.

Is "doing what you ask" considered a sharp edge? Network-related tools don't really have safeties, ever (your linux host will happily "ip rule add 0 blackhole" without confirmation). Every case of flowspec shenanigans in the news has been operator error.

Re: Level 3 Global Outage

#392

Post by dsr_ » Tue, Sep 01, 2020, 11:13 PM UTC

quickthrower2 wrote: post #338

jlgaddis wrote: post #284

> "Root Cause: An offending flowspec announcement prevented BGP from establishing correctly, impacting client services." -- That doesn't really explain the "stuck" routes in their RRs... maybe it'll make sense once we've gotten some more details...

This might be a silly question but is there such a thing as CI/CD for this sort of thing that may have caught the problem?

There are two aspects to this:

1. Is there syntax correctness checking available, so you don't push a config that breaks machines? Yes.

2. Is there a DWIM check available, so you can see the effect of the change before committing? No. That would require a complete model of, at a minimum, your entire network plus all directly connected networks -- that still wouldn't be complete, but it could catch some errors.

Re: Level 3 Global Outage

#393

Post by mrguyorama » Wed, Sep 02, 2020, 3:00 PM UTC

parliament32 wrote: post #391

kitteh wrote: post #325

Earlier quoted context omitted.

Flowspec strikes again. Its a super useful tool if you want to blast out an ACL across your network in seconds (using BGP) but it has a number of sharp edges. Several networks, including Cloudflare have learned what it can do. I've seen a few networks basically blackhole traffic or even lock themselves out of routers due to a poorly made Flowspec rules or a bug in the implementation.

Is "doing what you ask" considered a sharp edge? Network-related tools don't really have safeties, ever (your linux host will happily "ip rule add 0 blackhole" without confirmation). Every case of flowspec shenanigans in the news has been operator error.

It's possible that if a tool allows you to destroy everything with a single click, that tool (or maybe process) is bad