Live data from Hacker News

Hunting down the stuck BGP routes

blog.benjojo.co.uk

11–20 of 35 posts

Re: Hunting down the stuck BGP routes

#11

Interesting read. Its interesting that this is a big in the specification and not implementation since bgp is so old. We must not hit this case often

It's actually becoming more frequent due to BGP speakers being increasingly multithreaded. In the olden days, if you were overloaded, that also meant no more keepalives being sent. With the power of multithreading you can now simultaneously be overloaded and still send keepalives! :D

Re: Hunting down the stuck BGP routes

#12
post #6

Earlier quoted context omitted.

Is a protocol change necessary here? Keep alives are already sent... and they would be held up if the TCP window hit 0? At which point the BGP/TCP session can be terminated and re-established.

Like you I don't see how a change in protocol is requried, an update to the RFC to say something SHOULD time out the connection if the send window is zero. That said I haven't read the specs with a toothcomb and perhaps there's something about how you MUST NOT drop the connection if you're getting keepalives? Get Cisco and Juniper to implement it and that's 75% of LINX covered at least, I assume other exchanges have…

If you don't put it in the RFC then you'll end up with five different solutions to this problem from five different vendors, and a nice 5x5 matrix of new hilarious edge cases when these are talking to each other and something wonky is happening to the TCP session.

Re: Hunting down the stuck BGP routes

#13
From the article:

> With the current “default free zone” containing around 1,000,000 routes

Back in ~1998 I was tasked with building a route collector/looking glass machine for an internet exchange point (sadly defunct). I remember the day we switched the collector on and acquired "all the routes", there were ~98,000 of them, you could've knocked me over with a feather. It was like looking into the Total Perspective Vortex. Having been out of that game for many years now I'd no idea we were up to 1M routes...wow. One of the RIPE conferences I attended back then there was much concern about the rapidly increasing size of the global routing table and whether vendors could build hardware powerful enough to keep up.

For anyone interested the route collector was built on FreeBSD (3.0 I think) and Zebra[0].

And finally, what cracking blog, especially stuff like this:

https://blog.benjojo.co.uk/post/eve-online-bgp-internet

[0]: https://en.wikipedia.org/wiki/GNU_Zebra

Re: Hunting down the stuck BGP routes

#14
I wonder if a robust consensus algorithm might be a better investment than a timeout. I would imagine there are other bugs in BGP implementations so having a routing table that's going to trend towards eventual consistency regardless of the starting point might be a more robust solution than just focusing on this one corner case. Might be a more intrusive change though & hard to get middleware to roll out such a change?

Re: Hunting down the stuck BGP routes

#15

From the article: > With the current “default free zone” containing around 1,000,000 routes Back in ~1998 I was tasked with building a route collector/looking glass machine for an internet exchange point (sadly defunct). I remember the day we switched the collector on and acquired "all the routes", there were ~98,000 of them, you could've knocked me over with a feather. It was like looking into the Total Perspective…

Just looked at my router in docklands, 833,000 IPv4 routes, from 1.0.0.0/24 to 223.255.64.0/18

32,528 of them are in the 103.0/8 range, but on the other hand 21.0.0.0/8 is advertised once, no subnets at all. (same with 26, 28, 30, 33, 73. I don't have a route for 9.0.0.0/8 aside from 9.9.9.0/24.

Only 108,000 IPv6 routes.

Re: Hunting down the stuck BGP routes

#16
post #11

Interesting read. Its interesting that this is a big in the specification and not implementation since bgp is so old. We must not hit this case often

It's actually becoming more frequent due to BGP speakers being increasingly multithreaded. In the olden days, if you were overloaded, that also meant no more keepalives being sent. With the power of multithreading you can now simultaneously be overloaded and still send keepalives! :D

Ah yes; that's one of my favorites - health check returning 200s instantaneously; actual service is a black hole.

Re: Hunting down the stuck BGP routes

#17

I wonder if a robust consensus algorithm might be a better investment than a timeout. I would imagine there are other bugs in BGP implementations so having a routing table that's going to trend towards eventual consistency regardless of the starting point might be a more robust solution than just focusing on this one corner case. Might be a more intrusive change though & hard to get middleware to roll out such a chan…

Given the size and complexity of the Internet, it might be worth considering making BGP tolerant to Bizantine failures.

Re: Hunting down the stuck BGP routes

#18
post #11

Interesting read. Its interesting that this is a big in the specification and not implementation since bgp is so old. We must not hit this case often

It's actually becoming more frequent due to BGP speakers being increasingly multithreaded. In the olden days, if you were overloaded, that also meant no more keepalives being sent. With the power of multithreading you can now simultaneously be overloaded and still send keepalives! :D

> With the power of multithreading you can now simultaneously be overloaded and still send keepalives! :D

Now that's progress!

Reminds me of some of the naive comments in a few of the recent posts on HN about programming multithreading. It's not as easy as just adding more threads.

Or sending in more trains :) https://www.youtube.com/watch?v=-hyttagGsz0

Re: Hunting down the stuck BGP routes

#19
Thinking out loud: When I read the BGP spec, I got the feeling that it was optimized for reduced churn. As the Internet routing table size increased and increase in CPU power of routers was an uncertainty, the architects of the Internet wanted to avoid extra BGP exchanges.

However, now it seems like the Internet is facing new challenges and a different trade-off might make sense. Why not add a "valid until" attribute on each route? The originating router would have to re-announce a new route every 24 hours. Failure to propagate the update at any point would automatically withdraw it. Of course, re-announcing 1M routes every day might be a lot, but at this point it feels worth considering.

Re: Hunting down the stuck BGP routes

#20

Nice article on the basic functionalities of the Internet backbone. I really like the animation explaining this article with nice pictures. In short, BGP has a bug that potentially created a huge outage in August 2020. The proposed fix is to imrove the BGP protocol with a new feature. It's not easy because, it's the backbone of internet. Let's see where this will go.

Is a protocol change necessary here? Keep alives are already sent... and they would be held up if the TCP window hit 0? At which point the BGP/TCP session can be terminated and re-established.

The RFC proposes to change the BGP finite-state machine, not the protocol.
Post reply on HN