How and Why Swiftype Moved from EC2 to Real Hardware • Hacker News Board

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#11

Post by kovyrin » Mon, Mar 16, 2015, 5:19 PM UTC

ryanfitz wrote: post #7

kovyrin wrote: post #2

Author here. Happy to answer any questions.

Other than ec2, what AWS services were you using and how did you migrate those? At minimum, I'd guess you were using elb for load balancing, sqs for queues and elastic cache for redis.

We are still a loyal customer for some of their services. For example, we still use S3 for off-site backups and Route53 is still our primary DNS provider.

For load balancing we have moved to a Route53 (health checks and round-robin) + a group of nginx+haproxy+lua-based frontend boxes.

Everything else was either built in-house or used open-source components and wasn't really tied to EC2 infrastructure.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#12

Post by toomuchtodo » Mon, Mar 16, 2015, 5:19 PM UTC

mattbeckman wrote: post #5

"With Amazon we experienced networking issues, hanging VM instances, unpredictable performance degradation (probably due to noisy neighbors sharing our hardware, but there was no way to know) and numerous other problems. " Why do I get the feeling it was kind of a cop-out to just pack up and move without finding the root cause? I've seen it plenty of times: the "best solution" is to just find a different hosting prov…

> In my experience, I've never found an issue with an application on AWS that wasn't caused by either a misunderstanding of what was being offered (e.g. not provisioning enough PIOPS for database volumes), or simply issues with the application code.

You haven't been using Amazon long enough then.

Amazon is great for proof of concept. No upfront costs, extremely scalable, etc. Unfortunately, its expensive compared to physical hardware once you get to scale, and you may never solve underlying performance issues due to it being a shared tenant environment, even if you're a Netflix-sized customer.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#13

Post by jarjoura » Mon, Mar 16, 2015, 5:21 PM UTC

nanoGeek wrote: post #10

One question. If real hardware was always 50% cheaper then AWS wouldn't have been such a success. Can you please explain in which scenarios it makes economical sense to use AWS compared to real hardware?

AWS is always going to include a premium because they take care of the DevOps portion of your infrastructure. There are plenty of virtual hosting companies that cost significantly less than dedicated hardware, if you won't need all the bells and whistles.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#14

Post by fsniper » Mon, Mar 16, 2015, 5:23 PM UTC

Oh bare metal reloaded :)

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#15

Post by josh2600 » Mon, Mar 16, 2015, 5:24 PM UTC

toomuchtodo wrote: post #12

mattbeckman wrote: post #5

"With Amazon we experienced networking issues, hanging VM instances, unpredictable performance degradation (probably due to noisy neighbors sharing our hardware, but there was no way to know) and numerous other problems. " Why do I get the feeling it was kind of a cop-out to just pack up and move without finding the root cause? I've seen it plenty of times: the "best solution" is to just find a different hosting prov…

> In my experience, I've never found an issue with an application on AWS that wasn't caused by either a misunderstanding of what was being offered (e.g. not provisioning enough PIOPS for database volumes), or simply issues with the application code. You haven't been using Amazon long enough then. Amazon is great for proof of concept. No upfront costs, extremely scalable, etc. Unfortunately, its expensive compared to…

You can use abstraction layers to isolate yourself from issues with the underlying metal. For example, I had a good thread the last time the maintenance reboots happened: https://news.ycombinator.com/item?id=9120289.

Solving multi-tenancy issues is hard, but not impossible. I think it's a lot easier with live migration. If a box is giving you problems, just move the load to a new box while maintaining the same IP addressing.

With respect to cost, yes, AWS gets expensive at scale, but if you're at scale your servers are generally not your major cost center (it's usually payroll and licensing).

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#16

Post by robteix » Mon, Mar 16, 2015, 5:26 PM UTC

jarjoura wrote: post #13

nanoGeek wrote: post #10

One question. If real hardware was always 50% cheaper then AWS wouldn't have been such a success. Can you please explain in which scenarios it makes economical sense to use AWS compared to real hardware?

AWS is always going to include a premium because they take care of the DevOps portion of your infrastructure. There are plenty of virtual hosting companies that cost significantly less than dedicated hardware, if you won't need all the bells and whistles.

> AWS is always going to include a premium because they take care of the DevOps portion of your infrastructure.

But wouldn't that apply also to SoftLayer?

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#17

Post by kovyrin » Mon, Mar 16, 2015, 5:27 PM UTC

theg2 wrote: post #6

kovyrin wrote: post #2

Author here. Happy to answer any questions.

Using EC2 here for nearly 2 years and you mention I/O problems and instance outages 2-3 times a week. Which size instances were you running? I ask because other than the VM security updates, none our instances have these sort of issues and some of them have a VERY long life (not ideal we know). I understand the cost savings and the rest of the reasoning but in my experience EC2 isn't THAT unreliable.

Oh, I know what you're talking about. We too had some instances (actually, a lot of those) that would run for a year with no issues. The problems started around the time you tried to push EC2 instances beyond an "idle, handling some requests just to keep from falling asleep" state. Pushing IO (even with provisioned IOPS) caused random IO stalls, pushing CPU caused REALLY uneven performance, etc.

And the only solution provided by EC2 support was always to buy more instances to keep them cold and happy. The problems with that approach (just to name a few): the cost (for a young startup burning money on idle infrastructure like that is not very wise IMO) and the fact, that the time to design, develop and deploy scale-out approach for each of your backend services is the time you could have spent trying to build your product (again, startup-specific; you'll have to think about across-the-board 100% scalability at some point).

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#18

Post by moe » Mon, Mar 16, 2015, 5:31 PM UTC

nanoGeek wrote: post #10

One question. If real hardware was always 50% cheaper then AWS wouldn't have been such a success. Can you please explain in which scenarios it makes economical sense to use AWS compared to real hardware?

in which scenarios it makes economical sense to use AWS compared to real hardware?

There's three types of workloads that make sense to run on EC2:

a) Extremely spiky/seasonal loads (batch jobs, event/campaign traffic)

b) Loads that can be structured as to run entirely from spot-instances (worker-pools)

c) Loads so small that the markup versus rented/dedicated hardware just doesn't matter

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#19

Post by adrianpike » Mon, Mar 16, 2015, 5:32 PM UTC

mattbeckman wrote: post #5

"With Amazon we experienced networking issues, hanging VM instances, unpredictable performance degradation (probably due to noisy neighbors sharing our hardware, but there was no way to know) and numerous other problems. " Why do I get the feeling it was kind of a cop-out to just pack up and move without finding the root cause? I've seen it plenty of times: the "best solution" is to just find a different hosting prov…

I've seen and fought issues with hanging and/or stalling EC2 instances, and the decision was made to move to physical hardware - it was one of those tradeoff choices you have to make about paying N dollars to throw money at hardware vs. M dollars at person-hours to investigate.

It's definitely kicking the can down the road (eventually you have to build such that failing infrastructure is transparent to your eng team), but I still think it was the right decision at the time. YMMV obviously. :)

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#20

Post by gtrubetskoy » Mon, Mar 16, 2015, 5:33 PM UTC

This is a no-brainer if you've ever done anything at scale. The explanation is rather simple - hardware is always "on the premise", yours or Amazon's. Someone needs to swap drives, motherboards, man the networking gear, run cables, etc. Amazon doesn't really get a break on the hardware cost because 10,000 servers do not cost less per server than 100 servers (in fact they cost more as the volume goes up if you need them to be identical). When it comes to labor cost - if you have enough hardware for at least one full time datacenter tech, you're in the same boat as Amazon.

So you're paying Amazon to do the same work you would do otherwise - only you're subject to their rules and procedures and Amazon being a profitable business needs to mark their services up.