Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Route leak incident on January 22, 2026

blog.cloudflare.com

Post reply on HN
Print view View original

51–60 of 64 posts

Re: Route leak incident on January 22, 2026

#51

Post by vlovich123 »

arter45 wrote: post #47
vlovich123 wrote: post #46

Earlier quoted context omitted.

Yeah, but all of those basically boil down to “the next outage will look different from before” which fine but isn’t an actual solution IMO. My point is you want to do that and gradual rollouts that you don’t make permanent until you’ve observed the real world behavior if you want to prevent all future outages. This specific temporary rollout and automatic rollback also has the side effect that even if you don’t do a…

Rollback is of course useful when things go wrong (and by the way the routers CF use natively support rollback features). What I’m questioning is flapping as a structured way to carry out network changes. Even a slow flap can cause issues downstream. Imagine a router handling hundreds of thousands of routes. Its software has a memory leak so any route received increases its RAM usage. A slow flap may well bring that…

> Its software has a memory leak so any route received increases its RAM usage.

Surely you realize this as a weak reason but thought the argument against is that it’s my problem for someone else’s misbehaving software? I mean anyone sane in networking would treat this as not their problem (or at least work with the major providers for whom it is to make this possible).

However the strongest reason why I don’t buy this is that routes change regularly as a matter of course so changing a route forward and back is no different from changing it twice and so this bug would already be causing you issues and this is maybe a small percentage of extra advertisements.

> what you get is a lot of sessions with behavior1 and then switching to behavior2 and so on, which can cause high buffer utilization or packet drops.

Again, this explanation largely relies on FUD rather than concrete explanations. BGP routes change regularly and often. Such issues if they exist are already problems and briefly advertising a new route for a period of time as a dry run doesn’t alter those issues in any meaningful way. The problem is you’re treating “flap” as somehow magically different from any normal route change when it’s not really meaningfully so.

Re: Route leak incident on January 22, 2026

#52

Post by vlovich123 »

eqvinox wrote: post #48
vlovich123 wrote: post #8

I’m a huge fan of flapping when it’s really hard to do progressive rollouts. What this would mean here is you switch advertising the old and new routes back and forth automatically and this happens let’s say for 1 minute max before the old config is restored. Then a human looks at various metrics before they push a button to really make the new config permanent. It gives you a cheap way to preflight what will happen…

> It gives you a cheap way to preflight what will happen when you make a globally impacting config change. Your "1-minute flap" can propagate and trigger load on every single DFZ BGP router on the planet . That's not cheap. And 1 minute is too short to even propagate across carriers. There are all kinds of timers working to reduce previous point; your update can still be propagating half an hour later. It can also ch…

Ok. 5 minutes. The point is clearly there’s route changes happening globally already. It should not be that much extra work to add like 10% more route changes (again - you’d batch the new route advertisements in one cohort rather than updating each individual route back and forth).

And stuck routes are a problem but not one this would make worse since those routes would get stuck from normal changes anyway.

The propagation problem isn’t real because clearly most route advertisements that handle most of the traffic actually happen quickly. You shouldn’t care about the long tail - you want to minimize the risk of your new route. The old route being present isn’t a problem and the new route disappearing back to the old also shouldn’t be a problem UNLESS the new route was buggy in which case you wanted to rollback anyway.

TLDR: these don’t feel like risks unique to advertising and then undoing it given the route publishing already has to be handled anyway AND cloudflare is a major Tier 1 ISP and handles a good chunk of the entire internet’s traffic. This isn’t about a strategy for some random tier 2/3 ISP.

Re: Route leak incident on January 22, 2026

#54

Post by arter45 »

vlovich123 wrote: post #51
arter45 wrote: post #47

Earlier quoted context omitted.

Rollback is of course useful when things go wrong (and by the way the routers CF use natively support rollback features). What I’m questioning is flapping as a structured way to carry out network changes. Even a slow flap can cause issues downstream. Imagine a router handling hundreds of thousands of routes. Its software has a memory leak so any route received increases its RAM usage. A slow flap may well bring that…

> Its software has a memory leak so any route received increases its RAM usage. Surely you realize this as a weak reason but thought the argument against is that it’s my problem for someone else’s misbehaving software? I mean anyone sane in networking would treat this as not their problem (or at least work with the major providers for whom it is to make this possible). However the strongest reason why I don’t buy thi…

In the session scenario, I was talking about firewalls, not BGP routers (although, of course, you could have firewall features on a BGP router).

What I'm saying is, there are ways to validate and carry out network changes in a pretty robust way, including gradual rollout (if that's what you want) by using route or firewall rules priority or other mechanisms.

I keep being skeptical about this flapping strategy, but if this works in your setup, good for you.

Re: Route leak incident on January 22, 2026

#56

Post by redeeman »

tjwebbnorfolk wrote: post #37
dfajgljsldkjag wrote: post #2

We already have the tools to stop this from happening today. The problem is not the technology but the fact that companies do not want to work together to fix it. It is sad that we let the internet break because people are too slow to use the safety features we have.

If a bunch of big tech companies started collaborating/colluding to implement this, we'd just have a bunch of people on HN decrying the "centralization" of the internet concentrated in the hands of a few. This is decentralization in action. You have to take the good with the bad.

there are definitely ways to make the bad much less bad

Re: Route leak incident on January 22, 2026

#57

Post by _bernd »

arjie wrote: post #20

Based on the number of times I've seen these posted about they seem quite frequent[0]. If I'm being honest, the entire BGP system seems to be very fragile with a massive blast radius. I get that it's super 'core' so it's hard to fix, and that it comes from a time when the Internet was more 'cooperative' (in the protocol sense of the word) but are there any attempts at a successor or is it impossible to do so fundamen…

Here you go: https://en.wikipedia.org/wiki/Resource_Public_Key_Infrastruc...

Re: Route leak incident on January 22, 2026

#58

Post by eqvinox »

vlovich123 wrote: post #52
eqvinox wrote: post #48

Earlier quoted context omitted.

> It gives you a cheap way to preflight what will happen when you make a globally impacting config change. Your "1-minute flap" can propagate and trigger load on every single DFZ BGP router on the planet . That's not cheap. And 1 minute is too short to even propagate across carriers. There are all kinds of timers working to reduce previous point; your update can still be propagating half an hour later. It can also ch…

Ok. 5 minutes. The point is clearly there’s route changes happening globally already. It should not be that much extra work to add like 10% more route changes (again - you’d batch the new route advertisements in one cohort rather than updating each individual route back and forth). And stuck routes are a problem but not one this would make worse since those routes would get stuck from normal changes anyway. The propa…

> This isn’t about a strategy for some random tier 2/3 ISP.

That's not a constraint you mentioned in your original post.

> Ok. 5 minutes. The point is clearly there’s route changes happening globally already. It should not be that much extra work to add like 10% more route changes […]

I see you haven't had to deal with the operational reality of devices handling things they weren't quite designed for, and/or have been overdue for replacement, and/or were just designed to the limit to begin with. Good for you. But your solution would affect the entire internet.

If you're serious, you could try posting your suggestion to the NANOG or RIPE mailing lists. At the very least you'll probably learn a whole new set of expletives and curses… but I'd recommend against it.

Re: Route leak incident on January 22, 2026

#59

Post by eqvinox »

_bernd wrote: post #57
arjie wrote: post #20

Based on the number of times I've seen these posted about they seem quite frequent[0]. If I'm being honest, the entire BGP system seems to be very fragile with a massive blast radius. I get that it's super 'core' so it's hard to fix, and that it comes from a time when the Internet was more 'cooperative' (in the protocol sense of the word) but are there any attempts at a successor or is it impossible to do so fundamen…

Here you go: https://en.wikipedia.org/wiki/Resource_Public_Key_Infrastruc...

That, and ASPA, and https://manrs.org/
Post reply on HN