Live data from Hacker News

Route leak incident on January 22, 2026

blog.cloudflare.com

1–10 of 64 posts

Re: Route leak incident on January 22, 2026

#2
We already have the tools to stop this from happening today. The problem is not the technology but the fact that companies do not want to work together to fix it. It is sad that we let the internet break because people are too slow to use the safety features we have.

Re: Route leak incident on January 22, 2026

#3
> we pushed a change via our policy automation platform to remove the BGP announcements from Miami

Is there any way to test these changes against a simulation of real world routes? Including to ensure that traffic that shouldn’t hit Cloudflare servers, continues to resolve routes that don’t hit Cloudflare?

I have to imagine there’s academic research on how to simulate a fork of global BGP state, no? Surely there’s a tensor representation of the BGP graph that can be simulated on GPU clusters?

If there’s a meta-rule I think of when these incidents occur, it’s that configuration rules need change management, and change management is only as good as the level of automated testing. Just because code hasn’t changed doesn’t mean you shouldn’t test the baseline system behavior. And here, that means testing that the Internet works.

Re: Route leak incident on January 22, 2026

#4
post #3

> we pushed a change via our policy automation platform to remove the BGP announcements from Miami Is there any way to test these changes against a simulation of real world routes? Including to ensure that traffic that shouldn’t hit Cloudflare servers, continues to resolve routes that don’t hit Cloudflare? I have to imagine there’s academic research on how to simulate a fork of global BGP state, no? Surely there’s a…

I assume it's not possible unless you know the in-memory state of all the other gateway routers on the internet, no? You can know what they advertise, but that's not the same thing as a full description of their internal state and how they will choose to update if a route gets withdrawn.

Re: Route leak incident on January 22, 2026

#6
post #3

> we pushed a change via our policy automation platform to remove the BGP announcements from Miami Is there any way to test these changes against a simulation of real world routes? Including to ensure that traffic that shouldn’t hit Cloudflare servers, continues to resolve routes that don’t hit Cloudflare? I have to imagine there’s academic research on how to simulate a fork of global BGP state, no? Surely there’s a…

You can cross-reference RADB, the RIRs, and looking glass servers, and you'd find 3 different pictures of the internet.

Re: Route leak incident on January 22, 2026

#7
I've had to read the RCA a couple of times to (probably) get what happened, even if I'm reasonably familiar with BGP.

Basically, my understanding (simplified) is:

- they originally had a Miami router advertise Bogota prefixes (=subnets) to Cloudflare's peers. Essentially, Miami was handling Bogota's subnets. This is not an issue.

- because you don't normally advertise arbitrary prefixes via BGP, policies were used. These policies are essentially if/then statements, carrying out certain actions (advertise or not, add some tags or remove them,...) if some conditions are matched. This is completely normal.

- Juniper router configuration for this kind of policy is (simplifying):

set from

set from

set then

set then

...

- prior to the incident, CF changed its network so that Miami didn't have to handle Bogota subnets (maybe Bogota does it on its own, maybe there's another router somewhere else)

- the change aimed at removing the configurations on Miami which were advertising Bogota subnets

- the change implementation essentially removed all lines from all policies containing "from IP in the list of Bogota prefixes". This is somewhat reasonable, because you could have the same policy handling both Bogota and, say, Quito prefixes, so you just want to remove the Bogota part.

HOWEVER, there was at least one policy like this:

(Before)

set from is_internal(prefix) == True

set from prefix in bogota_prefix_list

set then advertise

(After)

set from is_internal(prefix) == True

set then advertise

Which basically means: if you have an internal prefix advertise it

- an "internal prefix" is any prefix that was not received by another BGP entity (autonomous system)

- BGP routers in Cloudflare exchange routes to one another. This is again pretty normal.

- As a result of this change, all routes received by Miami through some other Cloudflare router were readvertised by Miami

- the result is CF telling the Internet (more accurately, its peers) "hey, you know that subnet? Go ask my Miami router!"

- obviously, this increases bandwidth utilization and latency for traffic crossing the Miami router.

Re: Route leak incident on January 22, 2026

#8
I’m a huge fan of flapping when it’s really hard to do progressive rollouts. What this would mean here is you switch advertising the old and new routes back and forth automatically and this happens let’s say for 1 minute max before the old config is restored. Then a human looks at various metrics before they push a button to really make the new config permanent. It gives you a cheap way to preflight what will happen when you make a globally impacting config change.

Re: Route leak incident on January 22, 2026

#10

I’m a huge fan of flapping when it’s really hard to do progressive rollouts. What this would mean here is you switch advertising the old and new routes back and forth automatically and this happens let’s say for 1 minute max before the old config is restored. Then a human looks at various metrics before they push a button to really make the new config permanent. It gives you a cheap way to preflight what will happen…

I’m not sure this would be a good idea in this kind of change.

Flapping is bad in the networking world.

Flapping BGP routes, specifically, is bad because it can stress all BGP routers involved to the point where they can “go crazy”. Routes are explicitly advertised, so if you keep changing the routes, you are tasking the router CPU to process new stuff, discard it and process new stuff. In fact, BGP route flaps are specifically the focus of an entire RFC: https://datatracker.ietf.org/doc/html/rfc2439

More in general, a flapping link (on/off/on/off) can really mess with TCP.

Flapping in the networking world is not something you want to do intentionally.

Post reply on HN