Migrating from AWS to AWS • Hacker News Board

Re: Migrating from AWS to AWS

#11

Post by jeffbarr » Thu, Oct 23, 2014, 9:25 PM UTC

legohead wrote: post #9

We just migrated over to VPC as well, and came across a really weird "bug". We auto-scale EC2, and randomly when auto-scaling, the new server couldn't connect to memcache (ElastiCache). Note that when you migrate over to VPC you have to migrate everything -- launch new ElastiCache servers in VPC, EC2 servers, RDS servers, etc. Back to the bug.. I'd ssh into the EC2 server, and when I telnetted to memcache, it wouldn'…

I will make sure that this gets to the ElastiCache team! Please feel free to send more info my way; my email address can be found in my profile.

Re: Migrating from AWS to AWS

#12

Post by darkstar999 » Thu, Oct 23, 2014, 9:40 PM UTC

pbadger0 wrote: post #7

I wish Instagram engineering posts were all just collections of pictures with comments, with a different Instagram filter for each picture.

Dogfoodin' it to the extreme

Re: Migrating from AWS to AWS

#13

Post by mmmooo » Thu, Oct 23, 2014, 10:02 PM UTC

legohead wrote: post #9

We just migrated over to VPC as well, and came across a really weird "bug". We auto-scale EC2, and randomly when auto-scaling, the new server couldn't connect to memcache (ElastiCache). Note that when you migrate over to VPC you have to migrate everything -- launch new ElastiCache servers in VPC, EC2 servers, RDS servers, etc. Back to the bug.. I'd ssh into the EC2 server, and when I telnetted to memcache, it wouldn'…

This sounds like arp cache. And can be "fixed" by arping (forced arp responses without a who has)

Re: Migrating from AWS to AWS

#14

Post by wldcordeiro » Thu, Oct 23, 2014, 10:31 PM UTC

Is Instagram still using a modified version of Django 1.1 (if I recall that was the version they were on) or do they stay on the newest version now?

Re: Migrating from AWS to AWS

#15

Post by iancarroll » Fri, Oct 24, 2014, 12:18 AM UTC

ceejayoz wrote: post #5

Someone1234 wrote: post #4

I'm used to AWS pricing but I'm still a little fuzzy on how AWS VPC pricing works. It is $0.05 per VPN Connection-hour. But in this context what is a "VPN connection?" Do you literally just set up your cloud for "free" (aside from paying for instances, etc) and then only pay $0.05 for every hour you spend connected to the private cloud externally? Does the VPC have any external visibility aside from the VPN connectio…

VPC is free (and actually, new AWS accounts launch instances by default into a default VPC Amazon sets up for you). The $0.05 per VPN Connection-hour is only if you want to connect your own network to your VPC. http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_VP... > Does the VPC have any external visibility aside from the VPN connections? And if it does, what is stopping you just setting up your own VPN ser…

One would assume the VPN also has a higher network throughput then any instance under $30.

Re: Migrating from AWS to AWS

#16

Post by dantiberian » Fri, Oct 24, 2014, 6:00 AM UTC

This has to be pretty high up the list of "Most creative uses of Zookeeper". I'm really impressed with the ingenuity involved with building such a glorious hack.

Re: Migrating from AWS to AWS

#17

Post by cperciva » Fri, Oct 24, 2014, 8:55 AM UTC

legohead wrote: post #9

We just migrated over to VPC as well, and came across a really weird "bug". We auto-scale EC2, and randomly when auto-scaling, the new server couldn't connect to memcache (ElastiCache). Note that when you migrate over to VPC you have to migrate everything -- launch new ElastiCache servers in VPC, EC2 servers, RDS servers, etc. Back to the bug.. I'd ssh into the EC2 server, and when I telnetted to memcache, it wouldn'…

eventually we'd get one with the same Mac address but new internal IP address

This sounds like an EC2 bug, not an ElastiCache bug. EC2 uses "locally administered addresses" for MACs (as it should), but the administrator assigning MAC addresses is responsible for ensuring that addresses are not reused within a collision domain.

Re: Migrating from AWS to AWS

#18

Post by stevekemp » Fri, Oct 24, 2014, 10:51 AM UTC

mmmooo wrote: post #13

legohead wrote: post #9

We just migrated over to VPC as well, and came across a really weird "bug". We auto-scale EC2, and randomly when auto-scaling, the new server couldn't connect to memcache (ElastiCache). Note that when you migrate over to VPC you have to migrate everything -- launch new ElastiCache servers in VPC, EC2 servers, RDS servers, etc. Back to the bug.. I'd ssh into the EC2 server, and when I telnetted to memcache, it wouldn'…

This sounds like arp cache. And can be "fixed" by arping (forced arp responses without a who has)

Pretty much textbook symptoms of arp-caching.

Re: Migrating from AWS to AWS

#19

Post by matthavener » Fri, Oct 24, 2014, 12:02 PM UTC

mmmooo wrote: post #13

legohead wrote: post #9

We just migrated over to VPC as well, and came across a really weird "bug". We auto-scale EC2, and randomly when auto-scaling, the new server couldn't connect to memcache (ElastiCache). Note that when you migrate over to VPC you have to migrate everything -- launch new ElastiCache servers in VPC, EC2 servers, RDS servers, etc. Back to the bug.. I'd ssh into the EC2 server, and when I telnetted to memcache, it wouldn'…

This sounds like arp cache. And can be "fixed" by arping (forced arp responses without a who has)

Yup, EC2 instances should be issuing a gratuitous arp on startup. I've seen the same thing on subnets with a lot of DHCP churn due to constantly rebooting embedded devices.

Re: Migrating from AWS to AWS

#20

Post by giovannibajo1 » Fri, Oct 24, 2014, 1:51 PM UTC

The details of the migration are juicy and it's probably a good idea to do it anyway, but I can't stop thinking that the initial stumbling block (clashing of private IP addresses) wouldn't have happened with IPv6.