Author here. Happy to answer any questions.
Did you reserve your instances? Were you using current generation instances (c3, m3, etc). Did you try to take advantage of traffic patterns to scale up and down the number of instances you were running?
How and Why Swiftype Moved from EC2 to Real Hardware
141–150 of 194 posts
Re: How and Why Swiftype Moved from EC2 to Real Hardware
#142Earlier quoted context omitted.
> There's not much need for a fancy article on a fancy website in order to understand a key concept of cloud computing: I wish it were true, but plenty of companies are gripped by cloud fever. I've seen quite a few going down the route of charging into the cloud not because they've run the numbers and found it stacks up, but because they want to be in the cloud, and Amazon have some great marketing people.
From personal experience this is absolutely true. The initial cost to migrate and learn the cloud practice is unbelievable, but once you have the process in place running on Amazon CAN save some cost down the road, including the hours you need to replace hardware. You will build tools or use existing tools to create your infrastructure and operation process. A generous estimation to reach a level of maturity is 1.5 y…
"You can definitely save cost by subscribing to reserved instances, but the downside is you have to put down money upfront (...)"
This isn't required anymore. AWS introduced new Reserved Instances options few months ago, including a "no upfront" option which still gets you ~40% discount over on-demand prices.
https://aws.amazon.com/blogs/aws/simplified-reserved-instanc...
Re: How and Why Swiftype Moved from EC2 to Real Hardware
#143Earlier quoted context omitted.
> Your operational overhead will increase _a lot_. Be ready to hire on a lot of ops staff if you expect them to do anything but put out fires. And as you grow you'll need experts, people like network engineers. Why? What are you talking about? You are hiring servers you are no colo'ing them. Networking them is not your problem. Your responsibility still starts from a root prompt just there's no VM layer between that…
and if you have more than one? or more than one dc? somebody needs to connect it, or you will need a VPN. it's not that easy without Clouds if you need connected servers. We switched to AWS since connecting servers in a dc isn't as cheap as people think of.
We get free inter-DC connectivity for free with Softlayer. There really is no difference for us between connecting to a server within a DC or between datacenters (aside from added latency, of cource) – the same private address space, the same access rules (we see all of our machines in a private backend network from any datacenter).
Re: How and Why Swiftype Moved from EC2 to Real Hardware
#144Author here. Happy to answer any questions.
How do you handle spikes in request volume? That is, one of the nice things about working in the cloud with dynamic sizing is that your costs should only be relative to your average load, not necessarily your peak load. Given the size of Swiftype, and the fact that you back tons of individual sites (instead of being your own site), there might be enough variance in the sites you back so that your peak and average loa…
Honestly, just as it is with many SAAS companies at some scale, we do not need care about any specific customer's traffic anymore. Simply because we get so much traffic already from our existing customers, that none of the new customers could generate enough to cause any significant blip on the radar. If a customer comes to us with some specific requirements (like being able to index 100MM documents with some specific response time guarantees), we build dedicated pieces of infrastructure for them, load-test it all and provide those guarantees. All of the others are placed in their own pools which have enough capacity to handle 3-5x of ALL of our current traffic with no issues, so any single customer would not be able to generate enough load to cause problems.
And, as I mentioned in the article, we could always order new boxes for any of our clusters and get them online within a couple hours, so we are able to scale up pretty quickly if needed.
Re: How and Why Swiftype Moved from EC2 to Real Hardware
#145Author here. Happy to answer any questions.
Did you reserve your instances? Were you using current generation instances (c3, m3, etc). Did you try to take advantage of traffic patterns to scale up and down the number of instances you were running?
We had reserved instanced and regular ones, we did not see any patterns in stability issues between those. Re: instance types - I do not really remember which instances we were using to be honest. And as for the scaling up and down - we had a hard time keeping it all up as it was, we did not want to spend resources trying to make it work with constantly changing node pools (though I understand, that it would push us to building are more robust infrastructure able to handle random node outages, we had a business to build and wanted to focus on the product instead of creating a perfectly scalable application for an early stage application).
Re: How and Why Swiftype Moved from EC2 to Real Hardware
#146Earlier quoted context omitted.
Often times companies move because their organization doesn't deliver, and the hope is a cloud company will do a better job of it. If you have a great team, I firmly believe hosting yourself is far, far, far less expensive. If you have a terrible team, then cloud (hosting) is less expensive. Even if it was exactly the same cost, you're gaining by not having to have a staff to run it, and the costs of managing them, e…
If you're a small org with relatively small volume self-hosting doesn't make sense.
Can you elaborate why not ?
You can get some cheap dedicated hosting (ex: [1]) for a fraction of the price.
It's so cheap compared to AWS you can order a few spare ones and still come out cheaper than your one beefy AWS instance ?
The only way it doesn't make sense is, if you need to scale up and down very fast ?
[1] 60 euro/month: Quad-Core Haswell, 32 GB (non-ECC) RAM, 240Gb SSD @ https://www.hetzner.de
Re: How and Why Swiftype Moved from EC2 to Real Hardware
#147Earlier quoted context omitted.
in which scenarios it makes economical sense to use AWS compared to real hardware? There's three types of workloads that make sense to run on EC2: a) Extremely spiky/seasonal loads (batch jobs, event/campaign traffic) b) Loads that can be structured as to run entirely from spot-instances (worker-pools) c) Loads so small that the markup versus rented/dedicated hardware just doesn't matter
Yes, completely agree. Maybe I'd just add one more case here: some users are OK with locking themselves up to AWS by treating it as a platform from the day one and building on top of AWS database/queue/etc services. For those people using EC2 just to run the app code and replacing instances when they misbehave may be a good idea.
This really is the best scenario for AWS (and Azure is also heading this direction somewhat). We spend pennies on the dollar in utilization fees on their services vs what it would cost to implement, maintain, support and scale equivalent services on top of compute.
And to your point somewhere else in here, it is a hell of a thing to try and move away from that platform. Yeah, it's super easy to beat EC2 on cost of compute resources, but really if you're running everything yourself on top of compute at AWS then you're doing it wrong.
Re: How and Why Swiftype Moved from EC2 to Real Hardware
#148"2-3 serious outages a week" with EC2?? Details, please.
Re: How and Why Swiftype Moved from EC2 to Real Hardware
#149"2-3 serious outages a week" with EC2?? Details, please.
Those weren't outages for a specific instance. But from the whole pool of instances we were running 2-3 would have networking issues, random unexplained hangs requiring an instance restart, huge CPU performance drops, IO hangs, etc, etc.
Re: How and Why Swiftype Moved from EC2 to Real Hardware
#150Earlier quoted context omitted.
If you're a small org with relatively small volume self-hosting doesn't make sense.
Can you elaborate why not ? You can get some cheap dedicated hosting (ex: [1]) for a fraction of the price. It's so cheap compared to AWS you can order a few spare ones and still come out cheaper than your one beefy AWS instance ? The only way it doesn't make sense is, if you need to scale up and down very fast ? [1] 60 euro/month: Quad-Core Haswell, 32 GB (non-ECC) RAM, 240Gb SSD @ https://www.hetzner.de
Hetzner is fantastic but if you're not in Europe, what's your latency like?