Migrating From AWS to FB • Hacker News Board

Re: Migrating From AWS to FB

#51

Post by jordo37 » Fri, Jun 27, 2014, 1:33 AM UTC

aristus wrote: post #19

Called it, five years ago: http://www.web2expo.com/webexny2009/public/schedule/detail/9... Run in multiple clouds from day one. Take the pain. It gives you flexibility. Basic vendor management 101.

This seems like exactly the sort of advice that is great in retrospect once you reach scale but is effectively useless until you hit that point. There are far more important things for a startup to worry about (product market fit, retention, stability) than how to make the tech side of an acquisition easy.

Re: Migrating From AWS to FB

#52

Post by general_failure » Fri, Jun 27, 2014, 2:07 AM UTC

Any reason not to choose docker over lxc? Is it because fb data centers are already lxc friendly?

Re: Migrating From AWS to FB

#53

Post by sumbry » Fri, Jun 27, 2014, 2:07 AM UTC

serge2k wrote: post #20

aristus wrote: post #19

Called it, five years ago: http://www.web2expo.com/webexny2009/public/schedule/detail/9... Run in multiple clouds from day one. Take the pain. It gives you flexibility. Basic vendor management 101.

This article mentions nothing about latency.

We're undertaking a similar project and the latency is almost negligible. In fact the latency is lower bridging between classic and vpc in the same availability zone than between two classic availability zones.

Re: Migrating From AWS to FB

#54

Post by themartorana » Fri, Jun 27, 2014, 2:30 AM UTC

blackaspen wrote: post #45

mikeyk wrote: post #43

Earlier quoted context omitted.

Mike from IG here. VPC was barely a thing when we got on AWS (2010) and at the time not the default. I would definitely have done VPC from day 1 in hindsight, though.

Hindsight is 20/20. I think you guys did an exceptional job to tackling a really difficult problem (I've been in the same position, migrating EC2 to Datacenters) and we determined that EC2 -> VPC -> Datacenters is really the only way, and Neti solves it surprisingly well. Going forward, hope that acquired companies opened their AWS accounts late enough that Amazon forced them to use VPC.

We're small, comparatively - 20-30 servers max - and we need to get in to VPC for a new cluster that requires static internal IPs. (Reboot an EC2 Classic instance and you may get a different 10.x address.)

In any case, the migration is daunting even at our size, although our devops team size is 1. I do wish they had VPC when we started.

Re: Migrating From AWS to FB

#55

Post by nbm » Fri, Jun 27, 2014, 2:35 AM UTC

general_failure wrote: post #52

Any reason not to choose docker over lxc? Is it because fb data centers are already lxc friendly?

The existing Facebook deployment system supports running deployments within an LXC (and setting up cgroups, &c.) and was written well before docker was available.

Some background:

* http://www.slideshare.net/dotCloud/tupperware-containerized-...

Re: Migrating From AWS to FB

#56

Post by Rapzid » Fri, Jun 27, 2014, 4:52 AM UTC

toomuchtodo wrote: post #6

0x006A wrote: post #2

why not just create a vpn between the nodes with another private IP space and send your data through that?

"This task looked incredibly daunting on the face of it; we were running many thousands of instances in EC2, with new ones spinning up every day. In order to minimize downtime and operational complexity, it was essential that instances running in both EC2 and VPC seemed as if they were part of the same network. AWS does not provide a way of sharing security groups nor bridging private EC2 and VPC networks. The only w…

So I guess it was a bit of hyperbole. A straight forward solution to a straight forward problem.

Re: Migrating From AWS to FB

#57

Post by deftnerd » Fri, Jun 27, 2014, 5:54 AM UTC

justincormack wrote: post #10

agwa wrote: post #3

> The main blocker to this easy migration was that Facebook’s private IP space conflicts with that of EC2 IPv6 adoption could not happen soon enough.

Facebook supports ipv6, but maybe it is external only? I know Google is ipv6 internally too. Although as EC2 has no ipv6 maybe that was the blocking issue.

You can reach Facebook over IPv6 at https://www.v6.facebook.com

Interestingly enough, last week when they had their 45 minute outage worldwide with some kind of routing problem, it was still up and running on that address

Re: Migrating From AWS to FB

#58

Post by SushiMon » Fri, Jun 27, 2014, 6:23 AM UTC

I'm wondering if they got nailed by out-migration charges and how much that was. I assume a bunch of their images were in S3. Amazon charges a pretty penny to take things out.

Re: Migrating From AWS to FB

#59

Post by stephen_g » Fri, Jun 27, 2014, 6:43 AM UTC

deftnerd wrote: post #57

justincormack wrote: post #10

Earlier quoted context omitted.

Facebook supports ipv6, but maybe it is external only? I know Google is ipv6 internally too. Although as EC2 has no ipv6 maybe that was the blocking issue.

You can reach Facebook over IPv6 at https://www.v6.facebook.com Interestingly enough, last week when they had their 45 minute outage worldwide with some kind of routing problem, it was still up and running on that address

Facebook has AAAA records on the main https://facebook.com domain, so you don't need the v6 part anymore.

Re: Migrating From AWS to FB

#60

Post by shiftb » Fri, Jun 27, 2014, 7:04 AM UTC

I got an error page inside the Instagram mobile app a few days ago and was surprised to see Facebook server chrome around the error message.

I'm impressed out how fast they got this migration done, considering how massive the scale they operate at is.