Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating From AWS to FB

instagram-engineering.tumblr.com

Post reply on HN
Print view View original

81–90 of 94 posts

Re: Migrating From AWS to FB

#81

Post by teddyh »

mpasternacki wrote: post #70
vacri wrote: post #67

Earlier quoted context omitted.

I've had only a very limited exposure to IPv6, but it seemed to me that the slogan "DNS solves it for you!" doesn't really pan out. It solves it if you're on a well-set-up network and have your DNS up and running happily, but with the ad-hoc networks my [limited] experience has seen, it hasn't been trivial. Essentially, it means you have to run an interpreter service (the DNS) to understand the network - one more bit…

Won't mdns / zeroconf / avahi / however it's called this week work for ad-hoc networks? It surely does the trick in LAN. It won't work properly across routers, at least not out of the box (tried that when configuring Tinc VPN), but maybe this would be a good direction?

The word you probably wanted to use was “Zeroconf”.

“Zeroconf”¹ is a name for the sum of two interacting standards, namely mDNS”² and “DNS-SD”³. Avahi⁴ is a free software implementation (for Linux and BSD) for a service where programs can register Zeroconf services (name & port number) and have Avahi announce them on the network. The other major implementation of a daemon of this kind is from Apple, and it is called “Bonjour”⁵.

This often gets confused, so, again: Zeroconf = standard. mDNS and DNS-SD = component standards. Avahi = A specific free software implementation. Bonjour = A specific proprietary implementation.

1) http://zeroconf.org/

2) http://www.multicastdns.org/

3) http://www.dns-sd.org/

4) http://avahi.org/

5) https://www.apple.com/support/bonjour/

Re: Migrating From AWS to FB

#82

Post by hythloday »

stephen_g wrote: post #59
deftnerd wrote: post #57

Earlier quoted context omitted.

You can reach Facebook over IPv6 at https://www.v6.facebook.com Interestingly enough, last week when they had their 45 minute outage worldwide with some kind of routing problem, it was still up and running on that address

Facebook has AAAA records on the main https://facebook.com domain, so you don't need the v6 part anymore.

And the IPv6 address ends face:b00c:0:1 - hat tip to them.

Re: Migrating From AWS to FB

#83

Post by blakesmith »

It's usually the stateful stuff that proves challenging in big datacenter moves, but I don't see any mention of data copying, replication, or moving. How did you guys tackle the problems of keeping data in sync and doing a clean cutover?

Re: Migrating From AWS to FB

#84

Post by SushiMon »

ceejayoz wrote: post #73
SushiMon wrote: post #58

I'm wondering if they got nailed by out-migration charges and how much that was. I assume a bunch of their images were in S3. Amazon charges a pretty penny to take things out.

I'm confused by this. S3 GET requests are the cheapest request type, and getting the images out would just cost you the bandwidth involved. Maybe you're mixing things up with Glacier?

GET Requests are cheap But I was thinking of bandwidth costs to get things out of S3 entirely and do a complete outmigration. But the prices there have come down quite a bit since I last checked.

Re: Migrating From AWS to FB

#86

Post by Sami_Lehtinen »

stephen_g wrote: post #66
samstave wrote: post #47

Earlier quoted context omitted.

But I can recall ip4 addresses en mass in my head. I can't memorize ip6 addresses easily. Plus, I'd like to keep pockets of private IPs that are never accessible as routable targets. Can you explain why I wouldn't to do this or why I should evolve my understanding of ip6 better?

Why does it matter if an address is public or not? Whether the address is publicly routable has nothing to do with whether the host is accessible... Or don't you have firewalls?

You can also bind services to selected interfaces only. That's what I tell to people complaining about publicly routable addresses.

Re: Migrating From AWS to FB

#87

Post by ch »

I think this is the biggest takeaway from the article:

Plan to change just the bare minimum needed to support the new environment, and avoid the temptation of “while we’re here.”

Good engineering is knowing how to act with surgical precision when necessary. This is what allows a craft like programming to operate in the confines of a business.

Re: Migrating From AWS to FB

#88

Post by cooltrance »

It is amazing how a post like this could reach the front page of hacker news just because it comes from Instagram rather than for its technical relevance.

They mentioned Neti but didn't dig into details other than "a dynamic iptables manipulation daemon, written in Python, and backed by ZooKeeper." and they mentioned the ip blocker which is an issue on almost every migration.

Also taking into consideration that they didn't write a post in the past 10 months, I am sure that they can do it better.

Re: Migrating From AWS to FB

#89

Post by justizin »

blackaspen wrote: post #39
cbhl wrote: post #4

Earlier quoted context omitted.

Perhaps that would become a bottleneck at Instagram's scale?

I believe it becomes a bottleneck at the scale of like, 5 servers. This is a tough problem, Neti is a heck of a lot better than tons of VPN connections everywhere.

you could probably do hundreds of servers, but you would want to die.

Re: Migrating From AWS to FB

#90

Post by rhoml »

tomphoolery wrote: post #32

Has Facebook ever been public about the tools they use for deploying new machines onto bare metal with Chef? My company faces similar problems, albeit at a much smaller scale, but still...I'm wondering what they have in place of a tool such as http://theforeman.org (which is very coupled to Puppet).

I think they just said this https://www.youtube.com/watch?v=SYZ2GzYAw_Q
Post reply on HN