Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Route leak incident on January 22, 2026

blog.cloudflare.com

Post reply on HN
Print view View original

41–50 of 64 posts

Re: Route leak incident on January 22, 2026

#41

Post by wiether »

jacquesm wrote: post #11

That's like what, one major incident per month now, Nov 18, Dec 5, and now this one? I'll bet JGC can write his own ticket by now, but unretiring would be really bad optics. He's on the board though and still keeping a watchful eye. But a couple more of these and CFs reputation will be in the gutter.

My understanding of Cloudflare's history is that they built their reputation and their client base on some high quality products.

And instead on focusing on maintaining those, they decided to go for more money, first adding new features on their products (at the risk of breaking them) and then adding new products altogether in a move to start being an actual cloud provider.

Priorities shifted from the quality products to pushing features daily, and the person who built and maintained the good products probably left or have been assigned to shinier products, leaving the base to decay.

As a daily user, its quite frustrating to have a console that is getting far worse than AWS/Azure, and features that are more a POC than actual production-ready features.

Re: Route leak incident on January 22, 2026

#42

Post by jgrahamc »

jacquesm wrote: post #11

That's like what, one major incident per month now, Nov 18, Dec 5, and now this one? I'll bet JGC can write his own ticket by now, but unretiring would be really bad optics. He's on the board though and still keeping a watchful eye. But a couple more of these and CFs reputation will be in the gutter.

https://blog.cloudflare.com/fail-small-resilience-plan/

Re: Route leak incident on January 22, 2026

#43

Post by arter45 »

erredois wrote: post #19
arter45 wrote: post #7

I've had to read the RCA a couple of times to (probably) get what happened, even if I'm reasonably familiar with BGP. Basically, my understanding (simplified) is: - they originally had a Miami router advertise Bogota prefixes (=subnets) to Cloudflare's peers. Essentially, Miami was handling Bogota's subnets. This is not an issue. - because you don't normally advertise arbitrary prefixes via BGP, policies were used. T…

I am not very familiar with Juniper config, but this phrase summarizes it well. "This means we (AS13335) took the prefix received from Meta (AS32934), our peer, and then advertised it toward Lumen (AS3356), one of our upstream transit providers. " basically you should not receive a prefix from an eBGP session ( different AS) and advertize to an eBGP session. As they mention at the next steps, good use of communities…

Yes of course, but from a test perspective, this kind of mistake, given their configuration snippet and how they wrote the RCA, it seems to suggest they were simply diff-ing the initial and desired configs as any VCS would do (or, more likely, a Juniper “show|compare” command).

This didn’t catch the fact that removing that line essentially removed all conditions, allowing received routes to be re-advertised by the Miami router.

Communities are useful in this case, but this kind of thing could have happened with any kind of configuration.

Example:

(Before)

set firewall family inet filter FILTER NAME term TERM1 from source-address 10.10.10.1

set firewall family inet filter FILTER NAME term TERM1 from destination-port ssh

set firewall family inet filter FILTER NAME term TERM1 then discard

What happens when you remove references to 10.10.10.1, maybe because that IP is not blacklisted anymore? You’re simply removing one condition, leaving all ssh traffic to be discarded. That’s essentially what happened with the BGP outage, only here you have no BGP communities to save you.

That’s why I re-read the RCA, because this kind of incident is way more general than BGP-specific misconfigurations.

Re: Route leak incident on January 22, 2026

#44

Post by arter45 »

vlovich123 wrote: post #38
arter45 wrote: post #10

Earlier quoted context omitted.

I’m not sure this would be a good idea in this kind of change. Flapping is bad in the networking world. Flapping BGP routes, specifically, is bad because it can stress all BGP routers involved to the point where they can “go crazy”. Routes are explicitly advertised, so if you keep changing the routes, you are tasking the router CPU to process new stuff, discard it and process new stuff. In fact, BGP route flaps are s…

Ok. You can flap it slower and less frequently. The RFC you mentioned talks about timers on the order of a minute or so. So I would say advertise the new route for 1 minute and unconditionally restore for 10. Then only after that advertise for 2 minutes and restore for 10. There’s clearly some interval of on/off that isn’t a problem and that’s an effective way to evaluate the impact you of a deployed route change gra…

This specific outage is the equivalent of this scenario:

You have an if/then statement with N conditions in AND. You remove one condition, leaving the rest of the statement unchanged. What happens?

The answer is that if you remove one condition, your input (in this case routes) is more likely to match N-1 conditions than N, so more input is going to be processed according to the “then” clause.

The impact of course depends on the fact that these were BGP routes, advertised to the Internet,… but the problem itself is generic.

What can you do?

1) check this kind of if/then statements with special care, in order to analyze under which condition the input is processed by the “then” clause. This is exactly one of their followups [1]

2) consider adding “global”, catch-all policies acting as an additional safety net (if applicable)

3) test your changes not just syntactically. Set up a test environment with multiple routers, apply the configuration and see what happens.

[1] Adding automatic routing policy evaluation into our CI/CD pipelines that looks specifically for empty or erroneous policy terms

Re: Route leak incident on January 22, 2026

#46

Post by vlovich123 »

arter45 wrote: post #44
vlovich123 wrote: post #38

Earlier quoted context omitted.

Ok. You can flap it slower and less frequently. The RFC you mentioned talks about timers on the order of a minute or so. So I would say advertise the new route for 1 minute and unconditionally restore for 10. Then only after that advertise for 2 minutes and restore for 10. There’s clearly some interval of on/off that isn’t a problem and that’s an effective way to evaluate the impact you of a deployed route change gra…

This specific outage is the equivalent of this scenario: You have an if/then statement with N conditions in AND. You remove one condition, leaving the rest of the statement unchanged. What happens? The answer is that if you remove one condition, your input (in this case routes) is more likely to match N-1 conditions than N, so more input is going to be processed according to the “then” clause. The impact of course de…

Yeah, but all of those basically boil down to “the next outage will look different from before” which fine but isn’t an actual solution IMO.

My point is you want to do that and gradual rollouts that you don’t make permanent until you’ve observed the real world behavior if you want to prevent all future outages. This specific temporary rollout and automatic rollback also has the side effect that even if you don’t do any of the “hardening steps” outlined, your system will still prevent any kind of mistake you’ve made from rolling out and becoming more permanent. Like I said, the “flapping” parameters can be tuned however you want and you can aggregate updates into an automated “release train”. If you want you can do so with automated health metrics although it can be hard to implement automating validation that behavior before and after the route is “correct” (maybe trained ML models would be helpful here).

This is btw in many ways how Google releases code into production - they bundle a bunch of PRs into a giant “publish” step - if CI fails or anything in production fails, they automatically rollback the entire set of changes since they can’t know which part of the release went bad. It’s a huge hammer to solve any issue they didn’t account for.

Re: Route leak incident on January 22, 2026

#47

Post by arter45 »

vlovich123 wrote: post #46
arter45 wrote: post #44

Earlier quoted context omitted.

This specific outage is the equivalent of this scenario: You have an if/then statement with N conditions in AND. You remove one condition, leaving the rest of the statement unchanged. What happens? The answer is that if you remove one condition, your input (in this case routes) is more likely to match N-1 conditions than N, so more input is going to be processed according to the “then” clause. The impact of course de…

Yeah, but all of those basically boil down to “the next outage will look different from before” which fine but isn’t an actual solution IMO. My point is you want to do that and gradual rollouts that you don’t make permanent until you’ve observed the real world behavior if you want to prevent all future outages. This specific temporary rollout and automatic rollback also has the side effect that even if you don’t do a…

Rollback is of course useful when things go wrong (and by the way the routers CF use natively support rollback features). What I’m questioning is flapping as a structured way to carry out network changes.

Even a slow flap can cause issues downstream. Imagine a router handling hundreds of thousands of routes. Its software has a memory leak so any route received increases its RAM usage. A slow flap may well bring that router to a halt. Now you might say, “hey, this is not my fault”, but it is still something that could happen to your routers or your peers.

Another aspect is that network devices can get Terabits/s of traffic. Now, a router is mostly stateless, but if you do this flapping thing to a firewall, what you get is a lot of sessions with behavior1 and then switching to behavior2 and so on, which can cause high buffer utilization or packet drops.

So, yes, of course you “flap” (rollback) when things go wrong, but you probably don’t do it intentionally to test what’s going on in a network change.

Re: Route leak incident on January 22, 2026

#48

Post by eqvinox »

vlovich123 wrote: post #8

I’m a huge fan of flapping when it’s really hard to do progressive rollouts. What this would mean here is you switch advertising the old and new routes back and forth automatically and this happens let’s say for 1 minute max before the old config is restored. Then a human looks at various metrics before they push a button to really make the new config permanent. It gives you a cheap way to preflight what will happen…

> It gives you a cheap way to preflight what will happen when you make a globally impacting config change.

Your "1-minute flap" can propagate and trigger load on every single DFZ BGP router on the planet. That's not cheap.

And 1 minute is too short to even propagate across carriers. There are all kinds of timers working to reduce previous point; your update can still be propagating half an hour later. It can also change state for when you do it for real. And worst of all, BGP routes can get stuck. It's rare, but a real problem.

Re: Route leak incident on January 22, 2026

#49

Post by bflesch »

mschuster91 wrote: post #36
rkagerer wrote: post #34

Earlier quoted context omitted.

The older I get, the less I buy into "too big to fail" arguments. I now view it as "can't fail soon enough". The sooner it breaks down, the sooner something better will supplant it. This last sentiment holds true generally since organizations no longer subject to meaningful competition inevitably squat on their laurels and stop excelling at the things they used to be good at. We've seen it everywhere - Boeing, Google…

> The sooner it breaks down, the sooner something better will supplant it. That's not always possible, because the counterparty - aka threat actors - is always growing bigger, and you practically need to be the size of Cloudflare, Akamai or the Big 3 cloud providers to be able to weather attacks. You need to have big enough pipes to data centers and exchange points worldwide, otherwise any sufficiently motivated atta…

"Threat actor" is a relative definition, because for Italy the Cloudflare CEO was a "threat actor" who openly threatened availability of their systems.

Cloudflare knows they are just a glorified firewall + CDN that's why they desperately push into edge computing and getting these dozens of features.

Re: Route leak incident on January 22, 2026

#50

Post by roenxi »

rkagerer wrote: post #34
jacquesm wrote: post #31

Earlier quoted context omitted.

John left in April last year I think so it probably isn't directly related, so please take my comment in jest, but still it is worrisome, CF is in many ways 'too big to fail' and if this really becomes a regular thing it is going to cause a lot of people focused on their 'nines' to be pissed off. One thing to their credit though: BGP is full of complexity and it definitely isn't the first time that something like thi…

The older I get, the less I buy into "too big to fail" arguments. I now view it as "can't fail soon enough". The sooner it breaks down, the sooner something better will supplant it. This last sentiment holds true generally since organizations no longer subject to meaningful competition inevitably squat on their laurels and stop excelling at the things they used to be good at. We've seen it everywhere - Boeing, Google…

There was never much of an argument behind "too big to fail", it is generally a euphemism for upper-class welfare. In a more realist world, "too big to fail" is a mis-statement of "too risky to keep". Everything fails eventually and keeping incentives aligned relies on having a mechanism - failure - to flush out incompetents.
Post reply on HN