So I keep coming into situations where I think this is the problem that's occurring (a stuck route). While I'd certainly love to be able to diagnosis this, would it even matter? There's no recourse that I can take as an end user is there?
Hunting down the stuck BGP routes
21–30 of 35 posts
Re: Hunting down the stuck BGP routes
#22From the article: > With the current “default free zone” containing around 1,000,000 routes Back in ~1998 I was tasked with building a route collector/looking glass machine for an internet exchange point (sadly defunct). I remember the day we switched the collector on and acquired "all the routes", there were ~98,000 of them, you could've knocked me over with a feather. It was like looking into the Total Perspective…
Just looked at my router in docklands, 833,000 IPv4 routes, from 1.0.0.0/24 to 223.255.64.0/18 32,528 of them are in the 103.0/8 range, but on the other hand 21.0.0.0/8 is advertised once, no subnets at all. (same with 26, 28, 30, 33, 73. I don't have a route for 9.0.0.0/8 aside from 9.9.9.0/24. Only 108,000 IPv6 routes.
Crazy! I first started working with BGP in 1996. I remember there being under 30,000 routes... total.
Re: Hunting down the stuck BGP routes
#23I wonder if a robust consensus algorithm might be a better investment than a timeout. I would imagine there are other bugs in BGP implementations so having a routing table that's going to trend towards eventual consistency regardless of the starting point might be a more robust solution than just focusing on this one corner case. Might be a more intrusive change though & hard to get middleware to roll out such a chan…
There isn't actually a consensus to be formed on the Internet. Communities, local configuration, etc. cause BGP routers to make local decisions about routes to advertise and re-advertise that aren't going be part of a concensus.
Re: Hunting down the stuck BGP routes
#24From the article: > With the current “default free zone” containing around 1,000,000 routes Back in ~1998 I was tasked with building a route collector/looking glass machine for an internet exchange point (sadly defunct). I remember the day we switched the collector on and acquired "all the routes", there were ~98,000 of them, you could've knocked me over with a feather. It was like looking into the Total Perspective…
Just looked at my router in docklands, 833,000 IPv4 routes, from 1.0.0.0/24 to 223.255.64.0/18 32,528 of them are in the 103.0/8 range, but on the other hand 21.0.0.0/8 is advertised once, no subnets at all. (same with 26, 28, 30, 33, 73. I don't have a route for 9.0.0.0/8 aside from 9.9.9.0/24. Only 108,000 IPv6 routes.
Two very different things are going on to cause the smaller number of IPv6 routes.
One of course is that some Autonomous Systems don't advertise IPv6, either they have no globally routable IPv6 or they only achieve IPv6 via a tunnel and so that's only advertised via their tunnel provider.
But more important, many Autonomous Systems only need to advertise one prefix in IPv6 because it's big enough. Even if your needs grow, because we'd done this before and because IPv6 addresses are plentiful the allocations were deliberately sparse - so your RIR can give you the adjacent addresses, meaning you still only need one route entry for your larger space.
With IPv4 a provider may find itself advertising hundreds or even sometimes thousands of routes to the same Autonomous System since the addresses they need to advertise aren't contiguous.
Re: Hunting down the stuck BGP routes
#25I wonder if a robust consensus algorithm might be a better investment than a timeout. I would imagine there are other bugs in BGP implementations so having a routing table that's going to trend towards eventual consistency regardless of the starting point might be a more robust solution than just focusing on this one corner case. Might be a more intrusive change though & hard to get middleware to roll out such a chan…
Given the size and complexity of the Internet, it might be worth considering making BGP tolerant to Bizantine failures.
There isn't a really useful metric for failure, though. Not every prefix of every AS needs to be reachable from every other. Unlike consensus problems where everyone wants to agree on the same state it's sufficient for the Internet to be in a working state where each AS has enough routes that they care about, and BGP is pretty good at achieving that.
Re: Hunting down the stuck BGP routes
#26From the article: > With the current “default free zone” containing around 1,000,000 routes Back in ~1998 I was tasked with building a route collector/looking glass machine for an internet exchange point (sadly defunct). I remember the day we switched the collector on and acquired "all the routes", there were ~98,000 of them, you could've knocked me over with a feather. It was like looking into the Total Perspective…
Interesting historical perspective, thanks. And to put that growth into more recent perspective, in the last 7 years we have had a "512 K" day and "768 K" day. See:
https://cumulusnetworks.com/blog/768k-day-importance-adaptab...
Re: Hunting down the stuck BGP routes
#27Earlier quoted context omitted.
It's actually becoming more frequent due to BGP speakers being increasingly multithreaded. In the olden days, if you were overloaded, that also meant no more keepalives being sent. With the power of multithreading you can now simultaneously be overloaded and still send keepalives! :D
Ah yes; that's one of my favorites - health check returning 200s instantaneously; actual service is a black hole.
One of many problems that are only solvable with a software watchdog.
Re: Hunting down the stuck BGP routes
#28Thinking out loud: When I read the BGP spec, I got the feeling that it was optimized for reduced churn. As the Internet routing table size increased and increase in CPU power of routers was an uncertainty, the architects of the Internet wanted to avoid extra BGP exchanges. However, now it seems like the Internet is facing new challenges and a different trade-off might make sense. Why not add a "valid until" attribute…
I think 24 hours would be too slim. BGP is "routing by rumor" across a lot of AS's. I think a week would be more interesting allowing for route propagation to be a little slower. Obv this requires the spec really changing which in the case of BGP is unlikely to happen other than little tweaks
Re: Hunting down the stuck BGP routes
#29So I keep coming into situations where I think this is the problem that's occurring (a stuck route). While I'd certainly love to be able to diagnosis this, would it even matter? There's no recourse that I can take as an end user is there?
As an end user, not really. If you can get in touch with a NOC that understands the problem and is willing to listen to you, maybe. The problem is it can be pretty hard to find someone.
Re: Hunting down the stuck BGP routes
#30I wonder if a robust consensus algorithm might be a better investment than a timeout. I would imagine there are other bugs in BGP implementations so having a routing table that's going to trend towards eventual consistency regardless of the starting point might be a more robust solution than just focusing on this one corner case. Might be a more intrusive change though & hard to get middleware to roll out such a chan…
[deleted]