Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating From AWS to FB

instagram-engineering.tumblr.com

Post reply on HN
Print view View original

31–40 of 94 posts

Re: Migrating From AWS to FB

#31

Post by conorgil145 »

aristus wrote: post #19

Called it, five years ago: http://www.web2expo.com/webexny2009/public/schedule/detail/9... Run in multiple clouds from day one. Take the pain. It gives you flexibility. Basic vendor management 101.

> Run in multiple clouds from day one. Take the pain. It gives you flexibility. Basic vendor management 101.

While "taking the pain" may yield flexibility in the long run, the most important thing in the short run is making sure that you are building something that people want, listening to users, and iterating the tech side of things as quickly as possible. I suspect that most devs have enough trouble dealing with a single cloud provider and that trying to work with multiple would could a significant decrease in iteration speed. I think that approach would kill most startups because of the technical overhead incurred.

Re: Migrating From AWS to FB

#32

Post by tomphoolery »

Has Facebook ever been public about the tools they use for deploying new machines onto bare metal with Chef? My company faces similar problems, albeit at a much smaller scale, but still...I'm wondering what they have in place of a tool such as http://theforeman.org (which is very coupled to Puppet).

Re: Migrating From AWS to FB

#34

Post by NathanKP »

I wonder why Instagram wasn't using VPC in the first place. I've been using AWS for a startup for a few years now and I had our instances running in VPC from about the second month onward.

It's been one of the best architecture decisions I've ever made. At this point we only use one public IP address. (If direct access to a machine is needed then you can connect via VPN running on the one bastion host with the public IP address, and this gives your machine access to the local IP addresses of instances running inside the VPC.)

All the machines in our cluster are protected inside local VPC address space, with the access by the external world being ELB to expose public service endpoints like the API and website. I can't think of any good reason why you wouldn't be using VPC in the first place. Having public IP addresses for private machines sounds like a recipe for disaster if you ever accidentally miss a port in your security rules.

Re: Migrating From AWS to FB

#35

Post by EvanAnderson »

sliverstorm wrote: post #29
rdl wrote: post #9

Earlier quoted context omitted.

Right, but if everyone used IPv6 there would be no need to use non-routeable private IPs for anything, you could just use non-conflicting IPv6 addresses and not route them.

I kind of like having standard private subnets. My router is always 192.168.1.1, or sometimes 10.0.1.1, and so is my friend's, my parent's, and my grandparent's.

It creates a lot of pain when you need to connect these networks to each other, though. Admittedly, that's probably not a problem for parents, friends, etc.

At my Old Job I demanded we keep a "registry" of the RFC1918 address space we allocated to Customers. We never allocated Customers in overlapping address spaces. It made VPN connectivity to Customer A while on-site with Customer B much easier. It also helped out in one case where one Customer acquired another.

Re: Migrating From AWS to FB

#36

Post by maceip »

cheeseprocedure wrote: post #21
maceip wrote: post #17

"Facebook’s private IP space conflicts with that of EC2" ^ That wouldn't've happened in GCE (i.e., they should have been acquired by Google).

Are you sure? This document suggests GCE instances use the 10.x.x.x address space (just as AWS instances in EC2 Classic do): https://developers.google.com/compute/docs/instances-and-net...

""" Although Compute Engine doesn't allow creating an instance with a user-defined local IP address, you can use a combination of routes and an instance's ‑‑can_ip_forward ability to add local IP address as a network static address which then maps to your desired virtual machine instance.

For example, if you want to assign 10.1.1.1 specifically as a network address to a virtual machine instance, you can create a static network route that sends traffic from 10.1.1.1 to your instance, even if the instance's network address assigned by Compute Engine doesn't match your desired network address. """

Meaning they could have avoided conflicts using this mechanism.

Re: Migrating From AWS to FB

#37

Post by giovannibajo1 »

rbranson wrote: post #22
agwa wrote: post #18

Earlier quoted context omitted.

Amazon doesn't have to force customers to use their own IPv6 blocks. EC2 instances, whether inside or outside VPC, would be assigned unique IPv6 addresses from Amazon's address space (which would be extensive). If this were the norm when Instagram started out, it would have been just as easy to use, and there would have been no clash of addresses when migrating to Facebook.

Are you saying that Amazon would divide up it's own IPv6 address space and provide a subnet for each customer? That's really the only way it would work for our situation. I'm not sold that Amazon would actually have any motivation to go through the trouble to provide this as it would probably only impact 1% of their customers at best. The most rational path for them is to just assign addresses from a big pool and not…

Yes, Amazon should allocate a static /64 to each customer, possibly even more on (free) request. That would also make for very easy firewalling rules where you can whitelist connections from your own instances with just one simple firewall rule.

Re: Migrating From AWS to FB

#38

Post by giovannibajo1 »

DmitriRavinoff wrote: post #13
agwa wrote: post #3

> The main blocker to this easy migration was that Facebook’s private IP space conflicts with that of EC2 IPv6 adoption could not happen soon enough.

An EC2 rep has told me on more than one occasion that they have no plans to support ipv6 because the demand for it simply isn't there.

That's because people don't realize how useful it is in network ops because they re not used to it. This thread is an example were a glaring advantage of ipv6 wasn't immediately obvious.

Anybody who ever tried the VPC+ElasticIP+VPN braindeadness once should immediately file a feature request for ipv6; it's just that they don't probably think of it

Re: Migrating From AWS to FB

#39

Post by blackaspen »

cbhl wrote: post #4
0x006A wrote: post #2

why not just create a vpn between the nodes with another private IP space and send your data through that?

Perhaps that would become a bottleneck at Instagram's scale?

I believe it becomes a bottleneck at the scale of like, 5 servers.

This is a tough problem, Neti is a heck of a lot better than tons of VPN connections everywhere.

Re: Migrating From AWS to FB

#40

Post by derefr »

rbranson wrote: post #22
agwa wrote: post #18

Earlier quoted context omitted.

Amazon doesn't have to force customers to use their own IPv6 blocks. EC2 instances, whether inside or outside VPC, would be assigned unique IPv6 addresses from Amazon's address space (which would be extensive). If this were the norm when Instagram started out, it would have been just as easy to use, and there would have been no clash of addresses when migrating to Facebook.

Are you saying that Amazon would divide up it's own IPv6 address space and provide a subnet for each customer? That's really the only way it would work for our situation. I'm not sold that Amazon would actually have any motivation to go through the trouble to provide this as it would probably only impact 1% of their customers at best. The most rational path for them is to just assign addresses from a big pool and not…

> Are you saying that Amazon would divide up it's own IPv6 address space and provide a subnet for each customer?

That's what Linode does. Everyone gets a /64 to use as they please.

Post reply on HN