How and Why Swiftype Moved from EC2 to Real Hardware • Hacker News Board

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#101

Post by AstroChimpHam » Mon, Mar 16, 2015, 7:25 PM UTC

chx wrote: post #94

bobofettfett wrote: post #33

My 5cent: If you have lots and lots of money and a high margin business, do yourself a favor and go with Amazon (much less hassle with contract management and low level challenges). If you need to scale month to month and are growing 50% per month, go with Amazon. If you are very small and can live with 10 instances, go with Amazon. If CAPEX doesn't help you and for whatever reasons you need to spend OPEX, go with Am…

> If you need to scale month to month and are growing 50% per month, Then rent more servers. > If you are very small and can live with 10 instances, Then rent a few servers. I maintain that there are extremely few cases for a typical website to use the cloud. To handle peaks, it is both simpler and cheaper to keep enough capacity just idling around than spinning up and down Amazon instances. The cloud is almost alway…

Why rent more servers when it's so much easier to spin them up as you need them in AWS and tear them down (and no longer pay for them) when you don't. If you're expecting unknown change in scale, it's so much easier to be able to just spin up servers to keep up than anything else.

> To handle peaks, it is both simpler and cheaper to keep enough capacity just idling around than spinning up and down Amazon instances.

At a growing website, you have no idea how much "enough" is. Why try to estimate caps when you don't have to?

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#102

Post by merb » Mon, Mar 16, 2015, 7:27 PM UTC

rodgerd wrote: post #61

snoopybbt wrote: post #43

There's not much need for a fancy article on a fancy website in order to understand a key concept of cloud computing: Cloud computing offers you the great and awesome advantages of being able to instantly scale your application, replicate your data and basically just grow according to your business volume, and all this without significant investments, delivery time, setup time, people time, maintenance or anything bu…

> There's not much need for a fancy article on a fancy website in order to understand a key concept of cloud computing: I wish it were true, but plenty of companies are gripped by cloud fever. I've seen quite a few going down the route of charging into the cloud not because they've run the numbers and found it stacks up, but because they want to be in the cloud, and Amazon have some great marketing people.

That's cause you need to use the feature of your cloud. Run lots of servers and shoot them, when you dont need them. That's why the cloud is better than anything else. But most people just rent their xl3.large instance.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#103

Post by bobofettfett » Mon, Mar 16, 2015, 7:27 PM UTC

chx wrote: post #94

bobofettfett wrote: post #33

My 5cent: If you have lots and lots of money and a high margin business, do yourself a favor and go with Amazon (much less hassle with contract management and low level challenges). If you need to scale month to month and are growing 50% per month, go with Amazon. If you are very small and can live with 10 instances, go with Amazon. If CAPEX doesn't help you and for whatever reasons you need to spend OPEX, go with Am…

> If you need to scale month to month and are growing 50% per month, Then rent more servers. > If you are very small and can live with 10 instances, Then rent a few servers. I maintain that there are extremely few cases for a typical website to use the cloud. To handle peaks, it is both simpler and cheaper to keep enough capacity just idling around than spinning up and down Amazon instances. The cloud is almost alway…

From my experience renting more servers with 50% growth is a challenge. A lot of things go wrong when installing a lot of servers each month.

Also from my experience, with 10 instances the money you save with custom servers is negligible and contract and SLA management, multi datacenter etc. is easier with a cloud provider than renting servers. At least where I've rented servers in the past.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#104

Post by chx » Mon, Mar 16, 2015, 7:29 PM UTC

kovyrin wrote: post #98

chx wrote: post #89

Earlier quoted context omitted.

> Your operational overhead will increase _a lot_. Be ready to hire on a lot of ops staff if you expect them to do anything but put out fires. And as you grow you'll need experts, people like network engineers. Why? What are you talking about? You are hiring servers you are no colo'ing them. Networking them is not your problem. Your responsibility still starts from a root prompt just there's no VM layer between that…

Yes, it really does look like @stephen-mw is talking about a migration to an owned/colocated hardware. And that's a very different kind of hairy mess I'd prefer to stay away from as long as humanly possible.

OMG, colo. I've been the architect of a Top 500 website acquired by a Top 100 and even for that we just rented a few servers (you'd be surprised how few -- a dozen or so). I do not really know how big you need to be make colo a sensible choice but quite probably much, much bigger than most think of. We ran the numbers and they weren't pretty.

And for cloud: we purchased video conversion as a service, that one ran in the cloud. I can see how that makes sense.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#105

Post by merb » Mon, Mar 16, 2015, 7:34 PM UTC

chx wrote: post #89

lucas_alwin wrote: post #87

Here's my anecdotal, one-data-point experience from moving a giant EC2 environment to datacenter: 1. Your operational overhead will increase _a lot_. Be ready to hire on a lot of ops staff if you expect them to do anything but put out fires. And as you grow you'll need experts, people like network engineers. 2. Any weirdness you experienced with AWS infrastructure will be replaced with weirdness in your own environme…

> Your operational overhead will increase _a lot_. Be ready to hire on a lot of ops staff if you expect them to do anything but put out fires. And as you grow you'll need experts, people like network engineers. Why? What are you talking about? You are hiring servers you are no colo'ing them. Networking them is not your problem. Your responsibility still starts from a root prompt just there's no VM layer between that…

and if you have more than one? or more than one dc? somebody needs to connect it, or you will need a VPN. it's not that easy without Clouds if you need connected servers. We switched to AWS since connecting servers in a dc isn't as cheap as people think of.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#106

Post by kovyrin » Mon, Mar 16, 2015, 7:38 PM UTC

icedchai wrote: post #90

I'm convinced hybrid cloud is the way to go. Anything needing high IO performance should be on dedicated. Anything needed CPU/memory elasticity (worker nodes, etc) should be in the cloud. Assuming you can get low latency connectivity into AWS with DirectConnect, this might work?

Yeah, that (worker nodes, async processing, etc) seems like one of those ideal use-case for clouds that I could definitely support as a viable option for companies that aren't ready for all-in cloud deployment.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#107

Post by deitcher » Mon, Mar 16, 2015, 7:42 PM UTC

toomuchtodo wrote: post #97

deitcher wrote: post #95

Earlier quoted context omitted.

That is what I like about the cloud. Running in your own hardware lets you be, well, "lazy" about application architecture. Running in a place where it is shared and instances can disappear forces you to design a lot more robustly and nimbly. Of course, that design discipline is great wherever you are running....

Hardware is almost always cheaper than engineering time. Don't optimize prematurely.

Yes, but engineering is cheaper than running after real-time problems.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#108

Post by phil21 » Mon, Mar 16, 2015, 7:44 PM UTC

babo wrote: post #48

makmanalp wrote: post #27

Earlier quoted context omitted.

> The explanation is rather simple - hardware is always "on the premise", yours or Amazon's. Someone needs to swap drives, motherboards, man the networking gear, run cables, etc. > So you're paying Amazon to do the same work you would do otherwise - only you're subject to their rules and procedures and Amazon being a profitable business needs to mark their services up. But I thought that they were paying Softlayer to…

I would like to know the cost calculation after a year or two. With a handful of servers it's easy to get the false impression that HW failures are rare.

HW failures are rare. At least hardware failures that matter. Disks in a RAID set dying or redundant power supply failures are not critical, and even those are more rare than you would generally expect them to be. With a bit of standardization it's incredibly cheap to keep a pool of spares handy and RMA the failed components at a leisurely pace.

Plus, you're still engineering your applications to be just as fault-tolerant as if they were running in cloud, right? The only difference is you are not paying the virtualization overhead tax. A single server dying should leave you in a no less redundant state than a single VM dying. They should also be nearly as easily deployable.

This is based off my personal experience in datacenters with 5,000-10,000 installed servers. Anything other than a PSU or HDD failure is exceedingly rare.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#109

Post by kovyrin » Mon, Mar 16, 2015, 7:47 PM UTC

logn wrote: post #36

kovyrin wrote: post #2

Author here. Happy to answer any questions.

Did you seriously consider a traditional colo or other vendors like SoftLayer (e.g., Rackspace)? It seems like at some point your reasoning here will apply to a colo if you grow bigger.

Colo - that's an option I'll try to stay away from as long as it is humanly possible. All of my experiences with colo hardware caused a lifetime of pain so that I'm happy to be paying SL a premium for their outstanding services (I'm a huge fan of Softlayer as you have probably guessed).

Re: Rackspace and other providers – based on my real-life practical experience with a few of the largest providers in the States, SL quality of services and their provisioning speed are miles away from competitors could offer. So it was a no-brainer to go with SL and I'm happy we did.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#110

Post by amazon_not » Mon, Mar 16, 2015, 7:50 PM UTC

The biggest problem with AWS is the outrageous cost of bandwidth. Even if you ignore all the other cost differentials, the bandwidth charges will kill you at scale.

Unfortunately cloud computing, or at the very least AWS, overpromises and underdelivers at scale. All in all economically viable use cases for cloud computing are very few and very specific at scale.