Hunting down the stuck BGP routes
blog.benjojo.co.uk
Hunting down the stuck BGP routes
1–10 of 35 posts
Re: Hunting down the stuck BGP routes
#2Nice article on the basic functionalities of the Internet backbone. I really like the animation explaining this article with nice pictures. In short, BGP has a bug that potentially created a huge outage in August 2020. The proposed fix is to imrove the BGP protocol with a new feature. It's not easy because, it's the backbone of internet. Let's see where this will go.
Re: Hunting down the stuck BGP routes
#3Nice article on the basic functionalities of the Internet backbone. I really like the animation explaining this article with nice pictures. In short, BGP has a bug that potentially created a huge outage in August 2020. The proposed fix is to imrove the BGP protocol with a new feature. It's not easy because, it's the backbone of internet. Let's see where this will go.
Is a protocol change necessary here? Keep alives are already sent... and they would be held up if the TCP window hit 0? At which point the BGP/TCP session can be terminated and re-established.
Re: Hunting down the stuck BGP routes
#4Interesting read. Its interesting that this is a big in the specification and not implementation since bgp is so old. We must not hit this case often
Re: Hunting down the stuck BGP routes
#5Nice article on the basic functionalities of the Internet backbone. I really like the animation explaining this article with nice pictures. In short, BGP has a bug that potentially created a huge outage in August 2020. The proposed fix is to imrove the BGP protocol with a new feature. It's not easy because, it's the backbone of internet. Let's see where this will go.
Is a protocol change necessary here? Keep alives are already sent... and they would be held up if the TCP window hit 0? At which point the BGP/TCP session can be terminated and re-established.
I think the argument is if _your_ keep-alives are held up then currently you wait on _them_ terminating the session. If they are malicious or just not working well they may not do this.
Re: Hunting down the stuck BGP routes
#6Nice article on the basic functionalities of the Internet backbone. I really like the animation explaining this article with nice pictures. In short, BGP has a bug that potentially created a huge outage in August 2020. The proposed fix is to imrove the BGP protocol with a new feature. It's not easy because, it's the backbone of internet. Let's see where this will go.
Is a protocol change necessary here? Keep alives are already sent... and they would be held up if the TCP window hit 0? At which point the BGP/TCP session can be terminated and re-established.
Like you I don't see how a change in protocol is requried, an update to the RFC to say something SHOULD time out the connection if the send window is zero. That said I haven't read the specs with a toothcomb and perhaps there's something about how you MUST NOT drop the connection if you're getting keepalives?
Get Cisco and Juniper to implement it and that's 75% of LINX covered at least, I assume other exchanges have similar equipment makeup.
It seems reasonable behaviour to me.
It doesn't prevent the problem of the malicious BGP peer of course, but we know that already - if they choose to ignore your messages (while being happy with a high send-window) but continue to send keepalives you're equally screwed.
Re: Hunting down the stuck BGP routes
#7This reminds me of when YouTube was down for a lot of the world when Pakistan banned YouTube and one of the country's telecom company forgot to switch off their BGP route (if that is what the correct terminology would be).[0] Half as Interesting made a nice YouTube video about it.[1]
[0] https://www.cnet.com/news/how-pakistan-knocked-youtube-offli...
Re: Hunting down the stuck BGP routes
#8This reminds me of when YouTube was down for a lot of the world when Pakistan banned YouTube and one of the country's telecom company forgot to switch off their BGP route (if that is what the correct terminology would be).[0] Half as Interesting made a nice YouTube video about it.[1] [0] https://www.cnet.com/news/how-pakistan-knocked-youtube-offli... [1] https://www.youtube.com/watch?v=K9gnRs33NOk
This is less likely to happen these days as BGP routes are now validated through an open registry like RPKI https://www.arin.net/resources/manage/rpki/
I am not aware how popular and which company are using it, but I doubt that youtube is today as vulnerable as it was in 2008. BGP securities has a lot of tractions these days and is an interesting topic to follow.
Re: Hunting down the stuck BGP routes
#9Earlier quoted context omitted.
Is a protocol change necessary here? Keep alives are already sent... and they would be held up if the TCP window hit 0? At which point the BGP/TCP session can be terminated and re-established.
I think the argument is if _your_ keep-alives are held up then currently you wait on _them_ terminating the session. If they are malicious or just not working well they may not do this.
You can see the window size is zero though, so I think GP is suggesting sending a TCP reset or something similar.
Maybe this isn’t a good option because it would have too many undesirable side effects?
Re: Hunting down the stuck BGP routes
#10Nice article on the basic functionalities of the Internet backbone. I really like the animation explaining this article with nice pictures. In short, BGP has a bug that potentially created a huge outage in August 2020. The proposed fix is to imrove the BGP protocol with a new feature. It's not easy because, it's the backbone of internet. Let's see where this will go.
Is a protocol change necessary here? Keep alives are already sent... and they would be held up if the TCP window hit 0? At which point the BGP/TCP session can be terminated and re-established.
BGP Keepalives are not request-reply, they are simple scheduled transmissions. Which means even if one side is not reading, it may still be sending keepalives. So the other side keeps the session open, despite its own keepalives sitting in its send queue.
Also, any valid BGP message resets the keepalive timer, so the reading side just needs to occasionally pop something off the full queue and process it. Which, say, if you're swapping to hell and back, can still get done. (Assuming it even has the scheduling get to killing things due to holdtime expiry. It might just not be expiring anything anymore for reasons of floating face-down in the river.)