How and Why Swiftype Moved from EC2 to Real Hardware • Hacker News Board

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#131

Post by AndyNemmity » Mon, Mar 16, 2015, 8:28 PM UTC

jzawodn wrote: post #91

ChuckMcM wrote: post #79

Earlier quoted context omitted.

It is interesting coming at it from the other side, I get hit up all the time to move my infrastructure to the cloud, I give them the specs and what I pay per month, tell them that if they can beat it I'll switch. The closest anyone has come has been about 4x the cost I'm currently paying. But I note that I have a unique case that isn't covered by the cloud "architecture" (crawling the web and indexing it). Explainin…

Yeah, it's always fun when I tell someone at we're 100% self-hosted too. They never stop to think that traffic, load, and usage patterns on craigslist are pretty well understood at this point. Plus it's certainly cheaper than paying someone else to run all your hardware. You get to pick your peers, pick your hardware, and allocate as you see fit.

It's cheaper if you have a good team, which it sounds like you do.

A ton of places don't have a good team, and the cost numbers end up being a lot closer.

A great team will destroy Cloud margins. An Average-Bad team will fall in line, plus you don't have to run the team anymore.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#132

Post by Negitivefrags » Mon, Mar 16, 2015, 8:29 PM UTC

lucas_alwin wrote: post #87

Here's my anecdotal, one-data-point experience from moving a giant EC2 environment to datacenter: 1. Your operational overhead will increase _a lot_. Be ready to hire on a lot of ops staff if you expect them to do anything but put out fires. And as you grow you'll need experts, people like network engineers. 2. Any weirdness you experienced with AWS infrastructure will be replaced with weirdness in your own environme…

Contrast that to my experience with renting from Softlayer:

1) The operational overhead is the same as Amazon. Actually I think it's less because everything is so predictable. Every machine we get has identical performance to any other. We still don't have to care about the exact same list of things that EC2 provided but we also don't have to care about weird cloud issues.

2) See above answer but also Softlayer are happy to provide actual support for any issues in a timely manner. In general everything has been much more reliable that you actually have to think less about High Availabilty technologies that make your stack more complex. In the years I've been using them we have only had a few hard disk failures that were replaced in around an hour and we just failed over to a replicated slave manually for around 2-3 minutes of downtime.

3) Resources are no more finite than EC2. The only difference is that the provisioning time is 1 hour rather than 1 minute. That has still been perfectly fine to respond to unexpected load events.

4) Absolutely

5) Also great.

People often compare Cloud Hosting with Colo. What they should compare cloud hosting with is renting servers. You get 95% of the benefits of cloud with none of the drawbacks.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#133

Post by adevine » Mon, Mar 16, 2015, 8:29 PM UTC

kovyrin wrote: post #2

Author here. Happy to answer any questions.

How do you handle spikes in request volume? That is, one of the nice things about working in the cloud with dynamic sizing is that your costs should only be relative to your average load, not necessarily your peak load. Given the size of Swiftype, and the fact that you back tons of individual sites (instead of being your own site), there might be enough variance in the sites you back so that your peak and average load is pretty similar. For a single ecommerce site, though, they might get a huge peak over baseline if they do a big marketing push, for example. In that case, they might have to provision many more physical servers than they usually need to handle that peak. Just wanted to see if this issue came up in your planning.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#134

Post by rogerbinns » Mon, Mar 16, 2015, 8:30 PM UTC

toomuchtodo wrote: post #85

skwirl wrote: post #69

Earlier quoted context omitted.

>you're also paying Amazon's profits. Are you? Amazon is not known for posting profits, and is quite possibly operating AWS at a loss. They have promised to show financial results for AWS itself with the quarterly report being released next month, so we'll see.

AWS is hugely profitable: https://gigaom.com/2013/01/07/amazons-cloud-is-bigger-more-p...

A Motley Fool article claims that while AWS is successful, it isn't profitable, mainly because of the accounting behind capital expenditure. Last year Amazon spent more in interest than their entire operating income. http://www.fool.com/investing/general/2015/02/04/amazon-just...

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#135

Post by amazon_not » Mon, Mar 16, 2015, 8:34 PM UTC

wmf wrote: post #120

amazon_not wrote: post #110

The biggest problem with AWS is the outrageous cost of bandwidth. Even if you ignore all the other cost differentials, the bandwidth charges will kill you at scale. Unfortunately cloud computing, or at the very least AWS, overpromises and underdelivers at scale. All in all economically viable use cases for cloud computing are very few and very specific at scale.

Since people are sharing their experiences in this thread, has anyone tried using Direct Connect to get cheaper bandwidth?

There aren't a lot of savings to be had. In addition to paying for the Direct Connect infrastructure you still have to pay Amazon per GB charges, albeit at a slightly lesser rate.

The only real solution is to move the bandwidth usage off AWS.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#136

Post by emodendroket » Mon, Mar 16, 2015, 8:35 PM UTC

AndyNemmity wrote: post #129

rodgerd wrote: post #61

Earlier quoted context omitted.

> There's not much need for a fancy article on a fancy website in order to understand a key concept of cloud computing: I wish it were true, but plenty of companies are gripped by cloud fever. I've seen quite a few going down the route of charging into the cloud not because they've run the numbers and found it stacks up, but because they want to be in the cloud, and Amazon have some great marketing people.

Often times companies move because their organization doesn't deliver, and the hope is a cloud company will do a better job of it. If you have a great team, I firmly believe hosting yourself is far, far, far less expensive. If you have a terrible team, then cloud (hosting) is less expensive. Even if it was exactly the same cost, you're gaining by not having to have a staff to run it, and the costs of managing them, e…

If you're a small org with relatively small volume self-hosting doesn't make sense.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#137

Post by Negitivefrags » Mon, Mar 16, 2015, 8:36 PM UTC

bobofettfett wrote: post #103

chx wrote: post #94

Earlier quoted context omitted.

> If you need to scale month to month and are growing 50% per month, Then rent more servers. > If you are very small and can live with 10 instances, Then rent a few servers. I maintain that there are extremely few cases for a typical website to use the cloud. To handle peaks, it is both simpler and cheaper to keep enough capacity just idling around than spinning up and down Amazon instances. The cloud is almost alway…

From my experience renting more servers with 50% growth is a challenge. A lot of things go wrong when installing a lot of servers each month. Also from my experience, with 10 instances the money you save with custom servers is negligible and contract and SLA management, multi datacenter etc. is easier with a cloud provider than renting servers. At least where I've rented servers in the past.

I'm not sure where you see a difference between a VM and physical hardware when it comes to provisioning.

Sure, the physical hardware takes 1 hour rather than 1 minute to spin up, but the process is otherwise entirely identical.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#138

Post by Negitivefrags » Mon, Mar 16, 2015, 8:38 PM UTC

babo wrote: post #48

makmanalp wrote: post #27

Earlier quoted context omitted.

> The explanation is rather simple - hardware is always "on the premise", yours or Amazon's. Someone needs to swap drives, motherboards, man the networking gear, run cables, etc. > So you're paying Amazon to do the same work you would do otherwise - only you're subject to their rules and procedures and Amazon being a profitable business needs to mark their services up. But I thought that they were paying Softlayer to…

I would like to know the cost calculation after a year or two. With a handful of servers it's easy to get the false impression that HW failures are rare.

We have 100 physical servers and hardware failures really are very rare. Very rare.

In fact over 4 years we have only had 3 hard drives fail and no other hardware failures.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#139

Post by icedchai » Mon, Mar 16, 2015, 8:45 PM UTC

fleitz wrote: post #114

icedchai wrote: post #90

I'm convinced hybrid cloud is the way to go. Anything needing high IO performance should be on dedicated. Anything needed CPU/memory elasticity (worker nodes, etc) should be in the cloud. Assuming you can get low latency connectivity into AWS with DirectConnect, this might work?

Cpu also generally sucks due to cache thrashing on non-trivial datasets

Heh. So what is a good use case for EC2?

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#140

Post by kovyrin » Mon, Mar 16, 2015, 8:54 PM UTC

johnnycarcin wrote: post #71

Interesting article, thanks for sharing. We actually just went the exact opposite because of the larger scale issues we were having with Softlayer. Do you feel like you lost any resiliency by making the switch to physical servers (more virtual instances on one physical server, servers in the same rack, etc)?

No, I really do not think going to EC2 could be beneficial in any way in terms of improving resiliency compared to Softlayer. SL allows you to control which VLANs your box will end up on. VLANs could be treated as racks (since they do not allocate more than one VLAN per rack). Then you have multiple DCs in one region (e.g. DAL01, DAL05, DAL07, etc) and you have many different regions (DAL, SEA, WAS, AMS, etc).

I'd be very interested what problems you were having with them and at what scale. If this is a private topic, we could do it over email or some other medium if you like. You can contact me by any of the means listed here: http://kovyrin.net/contact/