How and Why Swiftype Moved from EC2 to Real Hardware • Hacker News Board

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#151

Post by hhw » Mon, Mar 16, 2015, 9:49 PM UTC

arjunnarayan wrote: post #47

hhw wrote: post #24

Earlier quoted context omitted.

No, there is significantly more complexity, overhead, and R&D to providing cloud services in comparison to bare metal. SoftLayer is actually a very expensive bare metal server provider. There are several good options that cost less than 1/3rd the price. Realistically, at just modest scale (a few physical servers), you should see 1/6th the cost of Amazon. The main benefits of Amazon is that it: a) allows you to scale…

Could you please list some competitors to SoftLayer? It's hard to get reliable opinions on cloud providers backed up with actual experience. I'd really appreciate it!

There's a plethora of different bare metal / dedicated server hosting providers, so recommending one is like recommending what type of car you should buy. The most important criteria generally involve: location, managed vs unmanaged, quality vs price, class of hardware, network uptime and hardware replacement SLA, number of servers, smaller or larger provider

The best resource to research different providers are the webhostingtalk.com forums. You can also contact me and I will do my best to advise you based on your desired criteria.

*Full Disclosure: I'm the founder/owner of a dedicated hosting company

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#152

Post by nasalgoat » Mon, Mar 16, 2015, 9:52 PM UTC

rodgerd wrote: post #61

snoopybbt wrote: post #43

There's not much need for a fancy article on a fancy website in order to understand a key concept of cloud computing: Cloud computing offers you the great and awesome advantages of being able to instantly scale your application, replicate your data and basically just grow according to your business volume, and all this without significant investments, delivery time, setup time, people time, maintenance or anything bu…

> There's not much need for a fancy article on a fancy website in order to understand a key concept of cloud computing: I wish it were true, but plenty of companies are gripped by cloud fever. I've seen quite a few going down the route of charging into the cloud not because they've run the numbers and found it stacks up, but because they want to be in the cloud, and Amazon have some great marketing people.

My old company moved, after I left, from a dedicated system I built that provided less than 1s response times to a cloud system that averaged 8s response times, because the company focus changed from maximizing performance to maximizing sizzle buzzwords.

I'm glad I left.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#153

Post by chx » Mon, Mar 16, 2015, 9:56 PM UTC

Negitivefrags wrote: post #137

bobofettfett wrote: post #103

Earlier quoted context omitted.

From my experience renting more servers with 50% growth is a challenge. A lot of things go wrong when installing a lot of servers each month. Also from my experience, with 10 instances the money you save with custom servers is negligible and contract and SLA management, multi datacenter etc. is easier with a cloud provider than renting servers. At least where I've rented servers in the past.

I'm not sure where you see a difference between a VM and physical hardware when it comes to provisioning. Sure, the physical hardware takes 1 hour rather than 1 minute to spin up, but the process is otherwise entirely identical.

1 hour? Welcome to 2015! OVH spins your server up in two minutes.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#154

Post by nasalgoat » Mon, Mar 16, 2015, 9:57 PM UTC

merb wrote: post #105

chx wrote: post #89

Earlier quoted context omitted.

> Your operational overhead will increase _a lot_. Be ready to hire on a lot of ops staff if you expect them to do anything but put out fires. And as you grow you'll need experts, people like network engineers. Why? What are you talking about? You are hiring servers you are no colo'ing them. Networking them is not your problem. Your responsibility still starts from a root prompt just there's no VM layer between that…

and if you have more than one? or more than one dc? somebody needs to connect it, or you will need a VPN. it's not that easy without Clouds if you need connected servers. We switched to AWS since connecting servers in a dc isn't as cheap as people think of.

Every enterprise router in existence has built-in tunnel support. Even the open source ones like PFSense.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#155

Post by nasalgoat » Mon, Mar 16, 2015, 10:01 PM UTC

kovyrin wrote: post #59

babo wrote: post #48

Earlier quoted context omitted.

I would like to know the cost calculation after a year or two. With a handful of servers it's easy to get the false impression that HW failures are rare.

Oh, there wasn't a handful of servers after we finished the migration (we have migrated a bit late IMO, so we had a lot of traffic even back then). And today, with much larger infrastructure, with hardware clusters specifically tailored to our customers needs, etc I'm pretty sure the same infrastructure on EC2 would cost more than 2x. (Update) Re: failures - with a ~50 servers we see a hardware issue (disk dead in a…

I ran several dozen Dell blade enclosures fully maxed out - well over 300 server blades - and in 3 years I had two disk failures, none of which were critical. Hardware is pretty reliable these days.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#156

Post by danieltillett » Mon, Mar 16, 2015, 10:02 PM UTC

hashtree wrote: post #124

A concrete example of how I saved a few hundred thousand dollars over the AWS by building quarter rack colocation setups with SuperMicro servers: https://gist.github.com/rockymadden/5561377 With Ansible, I spend no more than an hour a week, amortized, maintaining both the hardware and administration. I assume nodes for any specific role will fail, I only scale horizontally, I always have redundancy for every role, I…

Great resource. Are you going to keep this updated?

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#157

Post by nasalgoat » Mon, Mar 16, 2015, 10:04 PM UTC

icedchai wrote: post #90

I'm convinced hybrid cloud is the way to go. Anything needing high IO performance should be on dedicated. Anything needed CPU/memory elasticity (worker nodes, etc) should be in the cloud. Assuming you can get low latency connectivity into AWS with DirectConnect, this might work?

The speed of light is not your friend. You can never compete with the latency of a local network versus even the fastest connection to a cloud provider.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#158

Post by toast0 » Mon, Mar 16, 2015, 10:06 PM UTC

snoopybbt wrote: post #52

nanoGeek wrote: post #10

One question. If real hardware was always 50% cheaper then AWS wouldn't have been such a success. Can you please explain in which scenarios it makes economical sense to use AWS compared to real hardware?

Multiple reasons: 1) you can't get new hardware delivered and get it up and running, all in under 10 minutes. 2) also, assuming the previously gathered hardware is not needed anymore, you can't just return it and say "i used it only for two days because i had a traffic spike, take this 200$ and we're okay. 3) you can't programmatically install, configure, reinstall and reconfigure hardware configurations, networking…

1) ok got me there (but SoftLayer does offer VMs too, which presumably have a faster turn up time)

2) SL has hourly physical server rental now (turn up is quoted at 20-30 minutes though)

3) SL has an API for ordering changes, you can setup a script to run on first boot (and probably system images too). What are you thinking for network configuration? Really the only thing I've had to configure on SL is port speed (somewhat API accessible, but not if they need to drop in a 10G card/put you on a 10G rack, etc), and disabling the private ports (API accessible, real time changes).

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#159

Post by billyhoffman » Mon, Mar 16, 2015, 10:12 PM UTC

I used Rackspace (and later EC2) originally because of upfront capital costs. I bootstrapped my company, and paying a thousand dollars or more, plus hosting/racking fees, just wasn't doable. $50-100 bucks for a reasonable Windows server? Easy deal to make when cash flow is small.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#160

Post by guru_meditation » Mon, Mar 16, 2015, 10:23 PM UTC

snoopybbt wrote: post #43

There's not much need for a fancy article on a fancy website in order to understand a key concept of cloud computing: Cloud computing offers you the great and awesome advantages of being able to instantly scale your application, replicate your data and basically just grow according to your business volume, and all this without significant investments, delivery time, setup time, people time, maintenance or anything bu…

The reason why it is extremely hard to engineer robust large scale AWS cloud apps can be summarized under the umbrella of performance variance:

  - machine latency varies more, you can't control it

  - network latency varies more

  - storage latency varies more (S3, Redshift, etc.)

  - machine outages are more frequent

where more can be an order of magnitude more variation than on bare metal deployments. I am not saying the performance is that much worse, only that it will unpredictably vary for a certain instance. The interference is non gaussian and can happen in bursts as opposed to easy-to-model-and-anticipate white noise.

It's a lot harder to engineer cloud scale software to scale robustly and not degrade in latency when running on a large amount of nodes. For example, see [1]

Most of open-source cloud software does not come with these algorithms batteries included and it is not trivial to retrofit this kind of logic. Just being smart about loadbalancing won't cut it when at any given moment one of your nodes will become 10x slower than others even though your code is sound and in fact does not slow down like that.

In fact, what you lose in AWS convenience and "free" maintenance, you gain in simpler RPC/messaging/fault tolerance/storage infrastructure that can sometimes accommodate an order of magnitude more traffic or users on a machine then if deployed in AWS.

[1] http://research.google.com/people/jeff/latency.html