Playing Battleships over BGP (2018) • Hacker News Board

Re: Playing Battleships over BGP (2018)

#21

Post by JadeNB » Tue, Oct 05, 2021, 7:01 PM UTC

benjojo12 wrote: post #18

JadeNB wrote: post #16

Earlier quoted context omitted.

> It's really not the 2000's anymore, BGP is mostly safe and filtered. There are still improvements to be made (I've even written on the blog about them), but one persons immense fuck ups are far less likely to cause issues now that IRR filters and prefix limits exist. Any non-maliciously designed protocol probably can be used safely, but surely yesterday's events show that it is still eminently possible to use BGP d…

What part of yesterday was showing that it was possible to use BGP dangerously? If you are certain in this argument, then you master electric switch is dangerous because you could switch off the power to your house.

> If you are certain in this argument, then you master electric switch is dangerous because you could switch off the power to your house.

This seems like a response to an argument I haven't made yet! (All else aside, if you prebut my argument, it allows me not to make that argument.)

Sure, it's possible to do dangerous things with BGP; that alone is not why I say it's possible to use it dangerously. What is dangerous is the fact that a small and apparently innocent change can have such far-reaching consequences—for example, I'll bet there was no serious consideration at Facebook of not being able to open electronic door locks in the case of an apparently innocent BGP update.

I don't consider my master electric switch dangerous because I could switch off the power to my house. I would consider it dangerous if, after switching off the power to my house, I was ejected from my house, and could no longer open the doors of my house to get in and switch the power back on.

Re: Playing Battleships over BGP (2018)

#22

Post by midasuni » Tue, Oct 05, 2021, 7:15 PM UTC

INTPenis wrote: post #12

benjojo12 wrote: post #8

Earlier quoted context omitted.

Who said I tested non-live? The actual beta builds/sanity checks were done just with two VMs peered with each other, but the live internet one was done in one take (and never again, at least by me)

So how should these problems be mitigated? Have separate infrastructure for critical services or staging BGP or what?

It seems that the main problem Facebook group in restoring device was a lack of a completely separate out of band management network

If my network (way smaller than FB, but budget way lower) goes, I can get in via another ISP and WireGuard into the OOB network which is completly separate to the inband management.

Not every access switch is on OOB, but the core ones and a few critical devices are.

Re: Playing Battleships over BGP (2018)

#23

Post by HideousKojima » Tue, Oct 05, 2021, 8:19 PM UTC

JadeNB wrote: post #16

benjojo12 wrote: post #9

Earlier quoted context omitted.

To add on, BGP has a very much "meme" status of being scary and dangerous, and any touching will break youtube etc. [Mostly perpetuated by infosec circles] It's really not the 2000's anymore, BGP is mostly safe and filtered. There are still improvements to be made (I've even written on the blog about them), but one persons immense fuck ups are far less likely to cause issues now that IRR filters and prefix limits exi…

> It's really not the 2000's anymore, BGP is mostly safe and filtered. There are still improvements to be made (I've even written on the blog about them), but one persons immense fuck ups are far less likely to cause issues now that IRR filters and prefix limits exist. Any non-maliciously designed protocol probably can be used safely, but surely yesterday's events show that it is still eminently possible to use BGP d…

What happened yesterday was (appears to be) Facebook screwing up their own routing and DNS, not anyone else's. They didn't take down routing for any IPs and domains they didn't own. I can't imagine any other protocol making a mistake like FB's impossible

Re: Playing Battleships over BGP (2018)

#24

Post by HideousKojima » Tue, Oct 05, 2021, 8:23 PM UTC

JadeNB wrote: post #21

benjojo12 wrote: post #18

Earlier quoted context omitted.

What part of yesterday was showing that it was possible to use BGP dangerously? If you are certain in this argument, then you master electric switch is dangerous because you could switch off the power to your house.

> If you are certain in this argument, then you master electric switch is dangerous because you could switch off the power to your house. This seems like a response to an argument I haven't made yet! (All else aside, if you prebut my argument, it allows me not to make that argument.) Sure, it's possible to do dangerous things with BGP; that alone is not why I say it's possible to use it dangerously. What is dangerous…

>for example, I'll bet there was no serious consideration at Facebook of not being able to open electronic door locks in the case of an apparently innocent BGP update.

If that was actually the case, a lot of heads at FB should roll over this. The logic is simple and obvious, and if the sysadmins and network admins didn't think about this line of thinking then they're overpaid:

1) Our door control system is accessed via a public IP/address, not via an internal/private address.

2) Accessing our public IPs/addresses is dependent on BGP and DNS not getting borked.

Re: Playing Battleships over BGP (2018)

#25

Post by JadeNB » Tue, Oct 05, 2021, 9:07 PM UTC

HideousKojima wrote: post #24

JadeNB wrote: post #21

Earlier quoted context omitted.

> If you are certain in this argument, then you master electric switch is dangerous because you could switch off the power to your house. This seems like a response to an argument I haven't made yet! (All else aside, if you prebut my argument, it allows me not to make that argument.) Sure, it's possible to do dangerous things with BGP; that alone is not why I say it's possible to use it dangerously. What is dangerous…

>for example, I'll bet there was no serious consideration at Facebook of not being able to open electronic door locks in the case of an apparently innocent BGP update. If that was actually the case, a lot of heads at FB should roll over this. The logic is simple and obvious, and if the sysadmins and network admins didn't think about this line of thinking then they're overpaid: 1) Our door control system is accessed v…

> If that was actually the case, a lot of heads at FB should roll over this. The logic is simple and obvious, and if the sysadmins and network admins didn't think about this line of thinking then they're overpaid:

You say "if" as if it's a conditional, but surely the fact that it happened proves that no-one considered it (or, I suppose, that whoever did consider it didn't have enough sway to stop it from happening). There are, rightly, so many laws and regulations requiring that safe egress in case of emergency not be prevented, and I can't imagine anyone actually considering and tolerating even the slightest risk of an Internet issue preventing that egress. Well, I guess I can imagine lots of people doing lots of awful and harmful things, but I can't imagine anyone doing it in such a way that it would be this easy to get caught doing something blatantly illegal.

(Or were there safety measures in place that allowed egress, just not entry? I don't know the specifics, since my source is just the news stories that mention that the door locks didn't work and people couldn't get in—but maybe they could still get out?)