Migrating From AWS to FB • Hacker News Board

Re: Migrating From AWS to FB

#11

Post by agwa » Thu, Jun 26, 2014, 9:52 PM UTC

justincormack wrote: post #10

agwa wrote: post #3

> The main blocker to this easy migration was that Facebook’s private IP space conflicts with that of EC2 IPv6 adoption could not happen soon enough.

Facebook supports ipv6, but maybe it is external only? I know Google is ipv6 internally too. Although as EC2 has no ipv6 maybe that was the blocking issue.

Facebook makes extensive use of IPv6 internally. As you suggested, EC2's lack of support is the issue.

Edit: here are the slides from Facebook's presentation to the IPv6 World Congress in March about their internal IPv6 use. If IPv6 interests you, they're definitely worth a read: http://www.internetsociety.org/deploy360/wp-content/uploads/...

Re: Migrating From AWS to FB

#12

Post by toomuchtodo » Thu, Jun 26, 2014, 9:52 PM UTC

0x006A wrote: post #7

toomuchtodo wrote: post #6

Earlier quoted context omitted.

"This task looked incredibly daunting on the face of it; we were running many thousands of instances in EC2, with new ones spinning up every day. In order to minimize downtime and operational complexity, it was essential that instances running in both EC2 and VPC seemed as if they were part of the same network. AWS does not provide a way of sharing security groups nor bridging private EC2 and VPC networks. The only w…

Using a VPN would still be an option. Why write essentially your own VPN (neti) instead of using an existing VPN solution? VPC is not the only VPN you can use on EC2.

I believe Neti was a better solution at their scale (thousands of VMs, a dynamic production environment, etc).

Re: Migrating From AWS to FB

#13

Post by DmitriRavinoff » Thu, Jun 26, 2014, 9:57 PM UTC

agwa wrote: post #3

> The main blocker to this easy migration was that Facebook’s private IP space conflicts with that of EC2 IPv6 adoption could not happen soon enough.

An EC2 rep has told me on more than one occasion that they have no plans to support ipv6 because the demand for it simply isn't there.

Re: Migrating From AWS to FB

#14

Post by kawsper » Thu, Jun 26, 2014, 9:58 PM UTC

meritt wrote: post #5

Is neti open-source?

Managing iptables across datacenters and nodes would be a fun project to do with something like Serf (http://www.serfdom.io/)

Re: Migrating From AWS to FB

#15

Post by makomk » Thu, Jun 26, 2014, 10:00 PM UTC

rdl wrote: post #9

cheald wrote: post #8

Earlier quoted context omitted.

I think it has more to do with the fact that there are a standard series of private IP blocks.

Right, but if everyone used IPv6 there would be no need to use non-routeable private IPs for anything, you could just use non-conflicting IPv6 addresses and not route them.

Even if you did use non-routeable private IPs, IPv6 provides enough different private address ranges - more than there are individual addresses in IPv4 - that it's unlikely they'd conflict.

Re: Migrating From AWS to FB

#16

Post by rbranson » Thu, Jun 26, 2014, 10:02 PM UTC

agwa wrote: post #3

> The main blocker to this easy migration was that Facebook’s private IP space conflicts with that of EC2 IPv6 adoption could not happen soon enough.

Agree in principle, but it's unlikely for Amazon to ever force customers to allocate addresses from their own IPv6 block to begin with. Like many other startups, Instagram started with the bare-minimum setup outside of VPC, and scaled from there, not even knowing the benefits of VPC before it was too late for a simple cutover.

Re: Migrating From AWS to FB

#17

Post by maceip » Thu, Jun 26, 2014, 10:06 PM UTC

"Facebook’s private IP space conflicts with that of EC2"

^ That wouldn't've happened in GCE (i.e., they should have been acquired by Google).

Re: Migrating From AWS to FB

#18

Post by agwa » Thu, Jun 26, 2014, 10:09 PM UTC

rbranson wrote: post #16

agwa wrote: post #3

> The main blocker to this easy migration was that Facebook’s private IP space conflicts with that of EC2 IPv6 adoption could not happen soon enough.

Agree in principle, but it's unlikely for Amazon to ever force customers to allocate addresses from their own IPv6 block to begin with. Like many other startups, Instagram started with the bare-minimum setup outside of VPC, and scaled from there, not even knowing the benefits of VPC before it was too late for a simple cutover.

Amazon doesn't have to force customers to use their own IPv6 blocks. EC2 instances, whether inside or outside VPC, would be assigned unique IPv6 addresses from Amazon's address space (which would be extensive). If this were the norm when Instagram started out, it would have been just as easy to use, and there would have been no clash of addresses when migrating to Facebook.

Re: Migrating From AWS to FB

#19

Post by aristus » Thu, Jun 26, 2014, 10:10 PM UTC

Called it, five years ago: http://www.web2expo.com/webexny2009/public/schedule/detail/9...

Run in multiple clouds from day one. Take the pain. It gives you flexibility. Basic vendor management 101.

Re: Migrating From AWS to FB

#20

Post by serge2k » Thu, Jun 26, 2014, 10:12 PM UTC

aristus wrote: post #19

Called it, five years ago: http://www.web2expo.com/webexny2009/public/schedule/detail/9... Run in multiple clouds from day one. Take the pain. It gives you flexibility. Basic vendor management 101.

This article mentions nothing about latency.