Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Route leak incident on January 22, 2026

blog.cloudflare.com

Post reply on HN
Print view View original

31–40 of 64 posts

Re: Route leak incident on January 22, 2026

#31

Post by jacquesm »

stingraycharles wrote: post #29
jacquesm wrote: post #11

That's like what, one major incident per month now, Nov 18, Dec 5, and now this one? I'll bet JGC can write his own ticket by now, but unretiring would be really bad optics. He's on the board though and still keeping a watchful eye. But a couple more of these and CFs reputation will be in the gutter.

That’s what I also thought when I saw this incident. I wonder if there’s something up internally at Cloudflare or that it was always like this. I feel like something such as a route leak should not be something that happens to Cloudflare. I’m surprised they set their systems up to allow this human error.

John left in April last year I think so it probably isn't directly related, so please take my comment in jest, but still it is worrisome, CF is in many ways 'too big to fail' and if this really becomes a regular thing it is going to cause a lot of people focused on their 'nines' to be pissed off.

One thing to their credit though: BGP is full of complexity and it definitely isn't the first time that something like this goes wrong, it is just that at CF scale the impact is massive so there is no room for fuckups. But doing this sort of thing right 100% of the time is a really hard problem, and I'm happy I'm not in any way responsible for systems this important.

Whoever is responsible learned a lot of valuable lessons today (you hope).

Re: Route leak incident on January 22, 2026

#32

Post by patmorgan23 »

arjie wrote: post #20

Based on the number of times I've seen these posted about they seem quite frequent[0]. If I'm being honest, the entire BGP system seems to be very fragile with a massive blast radius. I get that it's super 'core' so it's hard to fix, and that it comes from a time when the Internet was more 'cooperative' (in the protocol sense of the word) but are there any attempts at a successor or is it impossible to do so fundamen…

There's several enhancements that have been strapped on to BGP over the years. The article talks about two at the end that will help reduce route leaks.

A wholesale protocol replacement is unlikely, but definitely more doable than replacing something like IP.

Re: Route leak incident on January 22, 2026

#34

Post by rkagerer »

jacquesm wrote: post #31
stingraycharles wrote: post #29

Earlier quoted context omitted.

That’s what I also thought when I saw this incident. I wonder if there’s something up internally at Cloudflare or that it was always like this. I feel like something such as a route leak should not be something that happens to Cloudflare. I’m surprised they set their systems up to allow this human error.

John left in April last year I think so it probably isn't directly related, so please take my comment in jest, but still it is worrisome, CF is in many ways 'too big to fail' and if this really becomes a regular thing it is going to cause a lot of people focused on their 'nines' to be pissed off. One thing to their credit though: BGP is full of complexity and it definitely isn't the first time that something like thi…

The older I get, the less I buy into "too big to fail" arguments. I now view it as "can't fail soon enough". The sooner it breaks down, the sooner something better will supplant it.

This last sentiment holds true generally since organizations no longer subject to meaningful competition inevitably squat on their laurels and stop excelling at the things they used to be good at. We've seen it everywhere - Boeing, Google, Microsoft (with OS's), etc.

Re: Route leak incident on January 22, 2026

#35

Post by vpShane »

jacquesm wrote: post #11

That's like what, one major incident per month now, Nov 18, Dec 5, and now this one? I'll bet JGC can write his own ticket by now, but unretiring would be really bad optics. He's on the board though and still keeping a watchful eye. But a couple more of these and CFs reputation will be in the gutter.

They made themselves 'Guardians of The Internet' then gave up. If they cared, these things wouldn't happen. How many more outages, accidents, incidents that effect millions of customers and millions of customers for other services are needed before they 'care'?

They don't, because at the end of the day it's not their problem, the money rolls in regardless.

It's sad, but it's how it is. If they cared, these things wouldn't happen. They have a lot of responsibility, but show none whatsoever.

Re: Route leak incident on January 22, 2026

#36

Post by mschuster91 »

rkagerer wrote: post #34
jacquesm wrote: post #31

Earlier quoted context omitted.

John left in April last year I think so it probably isn't directly related, so please take my comment in jest, but still it is worrisome, CF is in many ways 'too big to fail' and if this really becomes a regular thing it is going to cause a lot of people focused on their 'nines' to be pissed off. One thing to their credit though: BGP is full of complexity and it definitely isn't the first time that something like thi…

The older I get, the less I buy into "too big to fail" arguments. I now view it as "can't fail soon enough". The sooner it breaks down, the sooner something better will supplant it. This last sentiment holds true generally since organizations no longer subject to meaningful competition inevitably squat on their laurels and stop excelling at the things they used to be good at. We've seen it everywhere - Boeing, Google…

> The sooner it breaks down, the sooner something better will supplant it.

That's not always possible, because the counterparty - aka threat actors - is always growing bigger, and you practically need to be the size of Cloudflare, Akamai or the Big 3 cloud providers to be able to weather attacks. You need to have big enough pipes to data centers and exchange points worldwide, otherwise any sufficiently motivated attacker can just go and swamp them, but big pipes are helluvalot expensive so you need to have enough large and financially capable customers.

That's also why Cloudflare has expanded their offerings so much (e.g. Zero Trust), they need to have their infrastructure at some base load to economically justify it.

And that's also why Cloudflare will not be kicked off the throne any time soon. First of all, the initial costs to set up a competitor are absurdly high, second, how is a competitor supposed to lure large long term customers away from CF?

Any case, the real "fix" to Cloudflare being too-big-to-fail isn't building up competitors, it's getting the bad actors off of the Internet. Obviously that means holding both enemy (NK, Russia, China) and frenemy (India, Turkey) nations accountable, but it also means cleaning up shop at home - the aforementioned nation states and their botnet operators rely on an armada of hacked servers, ordinary computers and IoT devices in Western countries to carry out the actual work. And we clearly don't do anywhere near enough to get rid of these. I 'member a time when writing an abuse@ mail report that this would be taken seriously and the offender being disconnected by their ISP. These days, no one gives a fuck.

Re: Route leak incident on January 22, 2026

#37

Post by tjwebbnorfolk »

dfajgljsldkjag wrote: post #2

We already have the tools to stop this from happening today. The problem is not the technology but the fact that companies do not want to work together to fix it. It is sad that we let the internet break because people are too slow to use the safety features we have.

If a bunch of big tech companies started collaborating/colluding to implement this, we'd just have a bunch of people on HN decrying the "centralization" of the internet concentrated in the hands of a few.

This is decentralization in action. You have to take the good with the bad.

Re: Route leak incident on January 22, 2026

#38

Post by vlovich123 »

arter45 wrote: post #10
vlovich123 wrote: post #8

I’m a huge fan of flapping when it’s really hard to do progressive rollouts. What this would mean here is you switch advertising the old and new routes back and forth automatically and this happens let’s say for 1 minute max before the old config is restored. Then a human looks at various metrics before they push a button to really make the new config permanent. It gives you a cheap way to preflight what will happen…

I’m not sure this would be a good idea in this kind of change. Flapping is bad in the networking world. Flapping BGP routes, specifically, is bad because it can stress all BGP routers involved to the point where they can “go crazy”. Routes are explicitly advertised, so if you keep changing the routes, you are tasking the router CPU to process new stuff, discard it and process new stuff. In fact, BGP route flaps are s…

Ok. You can flap it slower and less frequently. The RFC you mentioned talks about timers on the order of a minute or so. So I would say advertise the new route for 1 minute and unconditionally restore for 10. Then only after that advertise for 2 minutes and restore for 10. There’s clearly some interval of on/off that isn’t a problem and that’s an effective way to evaluate the impact you of a deployed route change gradually over time rather than fucking up the internet for 25 minutes until someone figures out what’s going on.

And obviously you don’t do this on every individual route change - you batch them so it’s a release train.

If you think there’s better techniques other than “don’t break things” I’m all for it.

Re: Route leak incident on January 22, 2026

#39

Post by j16sdiz »

direwolf20 wrote: post #28
arjie wrote: post #25

Earlier quoted context omitted.

I see. That makes sense.

A route leak is often like this: an ISP in Pakistan is ordered to censor YouTube, so they add a route internally to YouTube's IP addresses that passes to their censoring machine, or to nowhere. They accidentally have their edge routers configured to pass this route to all their connected networks instead of keeping it internally to themselves. Some of their peers recognize this as the shortest route to YouTube and in…

sometimes it is just innocent:

An isp have lease a new 10Gb fiber to youtube for my own customers, the route is leaked to my peer and now every isp in the whole country is using my fiber for youtube.

Re: Route leak incident on January 22, 2026

#40

Post by re-thc »

stingraycharles wrote: post #29
jacquesm wrote: post #11

That's like what, one major incident per month now, Nov 18, Dec 5, and now this one? I'll bet JGC can write his own ticket by now, but unretiring would be really bad optics. He's on the board though and still keeping a watchful eye. But a couple more of these and CFs reputation will be in the gutter.

That’s what I also thought when I saw this incident. I wonder if there’s something up internally at Cloudflare or that it was always like this. I feel like something such as a route leak should not be something that happens to Cloudflare. I’m surprised they set their systems up to allow this human error.

> or that it was always like this

The focus has been on new features and moving fast for quite some years vs reliability.

Post reply on HN