Live data from Hacker News

Migrating From AWS to FB

instagram-engineering.tumblr.com

41–50 of 94 posts

Re: Migrating From AWS to FB

#41
post #8
post #3

> The main blocker to this easy migration was that Facebook’s private IP space conflicts with that of EC2 IPv6 adoption could not happen soon enough.

I think it has more to do with the fact that there are a standard series of private IP blocks.

You can always create multiple routing instances (namespaces) for overlapping IPv4 blocks with a VRF-based strategy. This would create separate FIBs and IMHO is a cleaner approach than mangling packets with iptables.

Re: Migrating From AWS to FB

#42
post #9
post #8

Earlier quoted context omitted.

I think it has more to do with the fact that there are a standard series of private IP blocks.

Right, but if everyone used IPv6 there would be no need to use non-routeable private IPs for anything, you could just use non-conflicting IPv6 addresses and not route them.

That's basically what a ULA is for though - see RFC 4193. If I'm not big enough to acquire my own allocations from ARIN, I'd prefer not to have to renumber every piece of equipment (even those not externally accessible) when changing ISPs.

Re: Migrating From AWS to FB

#43

I wonder why Instagram wasn't using VPC in the first place. I've been using AWS for a startup for a few years now and I had our instances running in VPC from about the second month onward. It's been one of the best architecture decisions I've ever made. At this point we only use one public IP address. (If direct access to a machine is needed then you can connect via VPN running on the one bastion host with the public…

Mike from IG here. VPC was barely a thing when we got on AWS (2010) and at the time not the default. I would definitely have done VPC from day 1 in hindsight, though.

Re: Migrating From AWS to FB

#45
post #43

I wonder why Instagram wasn't using VPC in the first place. I've been using AWS for a startup for a few years now and I had our instances running in VPC from about the second month onward. It's been one of the best architecture decisions I've ever made. At this point we only use one public IP address. (If direct access to a machine is needed then you can connect via VPN running on the one bastion host with the public…

Mike from IG here. VPC was barely a thing when we got on AWS (2010) and at the time not the default. I would definitely have done VPC from day 1 in hindsight, though.

Hindsight is 20/20.

I think you guys did an exceptional job to tackling a really difficult problem (I've been in the same position, migrating EC2 to Datacenters) and we determined that EC2 -> VPC -> Datacenters is really the only way, and Neti solves it surprisingly well.

Going forward, hope that acquired companies opened their AWS accounts late enough that Amazon forced them to use VPC.

Re: Migrating From AWS to FB

#46

I wonder why Instagram wasn't using VPC in the first place. I've been using AWS for a startup for a few years now and I had our instances running in VPC from about the second month onward. It's been one of the best architecture decisions I've ever made. At this point we only use one public IP address. (If direct access to a machine is needed then you can connect via VPN running on the one bastion host with the public…

If you assume they had no pressing need for any VPC specific functionality, you can get similar security by locking your security group/s down to only ELB for public service ports and having one instance in another security group with ssh/vpn allowed (to specific ips) as a jump box/vpn. Spending weeks of multiple teams engineering time to move to VPC without a pressing need would seem to me to make little business sense.

Re: Migrating From AWS to FB

#47
post #9
post #8

Earlier quoted context omitted.

I think it has more to do with the fact that there are a standard series of private IP blocks.

Right, but if everyone used IPv6 there would be no need to use non-routeable private IPs for anything, you could just use non-conflicting IPv6 addresses and not route them.

But I can recall ip4 addresses en mass in my head. I can't memorize ip6 addresses easily. Plus, I'd like to keep pockets of private IPs that are never accessible as routable targets.

Can you explain why I wouldn't to do this or why I should evolve my understanding of ip6 better?

Re: Migrating From AWS to FB

#48
post #23

Earlier quoted context omitted.

Are you saying that Amazon would divide up it's own IPv6 address space and provide a subnet for each customer? That's really the only way it would work for our situation. I'm not sold that Amazon would actually have any motivation to go through the trouble to provide this as it would probably only impact 1% of their customers at best. The most rational path for them is to just assign addresses from a big pool and not…

>The most rational path for them is to just assign addresses from a big pool and not bother with the all the fancy subnetting. That's what he's suggesting, though (I think). Except because it's IPv6, Amazon's big pool of addresses would never conflict with Facebook's big pool of addresses.

One thing that may not have come across in the post is that one of the reasons we moved into VPC first is that Direct Connect is basically just a big dump pipe to AWS without VPC in place. Without VPC, there's no way to advertise routes for just your instances or to ensure that only traffic to/from your instances goes across the Direct Connect.

Re: Migrating From AWS to FB

#49
post #36

Earlier quoted context omitted.

Are you sure? This document suggests GCE instances use the 10.x.x.x address space (just as AWS instances in EC2 Classic do): https://developers.google.com/compute/docs/instances-and-net...

""" Although Compute Engine doesn't allow creating an instance with a user-defined local IP address, you can use a combination of routes and an instance's ‑‑can_ip_forward ability to add local IP address as a network static address which then maps to your desired virtual machine instance. For example, if you want to assign 10.1.1.1 specifically as a network address to a virtual machine instance, you can create a stat…

Maintaining thousands of forwarding/routing configs sounds just as nasty as implementing Neti.

At any rate, Instagram's been around since 2010 and GCE didn't exist until June 2012 (and wasn't generally available until this past December).

Re: Migrating From AWS to FB

#50
post #9

Earlier quoted context omitted.

Right, but if everyone used IPv6 there would be no need to use non-routeable private IPs for anything, you could just use non-conflicting IPv6 addresses and not route them.

But I can recall ip4 addresses en mass in my head. I can't memorize ip6 addresses easily. Plus, I'd like to keep pockets of private IPs that are never accessible as routable targets. Can you explain why I wouldn't to do this or why I should evolve my understanding of ip6 better?

IPv6 address don't have to be long or publicly routable. ULA addresses (fd00::/8) are the IPv6 analogue to RFC1918 and you could theoretically use ULA addresses as short as fd00::1, fd00::2, fd00::3, and so on. Of course if you do this you run the risk of colliding with other people, so you're encouraged to randomly generate the next 40 bits after fd, which leaves you with addresses like fd32:5e26:381d::1. That's longer than IPv4 addresses, but it's a pretty fair tradeoff to get a globally unique address.

Even non-ULA IPv6 addresses need not be long. 2600:3c00:e000:6c::1 is the address of my server over at Linode, and I don't find that bad at all.

Post reply on HN