Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How and Why Swiftype Moved from EC2 to Real Hardware

highscalability.com

Post reply on HN
Print view View original

31–40 of 194 posts

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#31

Post by nanoGeek »

kovyrin wrote: post #25
nanoGeek wrote: post #10

One question. If real hardware was always 50% cheaper then AWS wouldn't have been such a success. Can you please explain in which scenarios it makes economical sense to use AWS compared to real hardware?

In my personal opinion (based on some real experience) EC2 for a startup makes a lot of sense for prototyping your application and your infrastructure (when you don't really know what is it you're building and what components you're building it from). At this stage you just is it as an easy way to get a set of Linux computers connected to the Internet. When you get to a point where you feel like this whole thing is g…

Great point. Also, on AWS if you use Amazon Linux you are even more 'locked-in'. Maybe that's the reason many hosting companies give you discounts at the beginning.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#32

Post by samstave »

kovyrin wrote: post #2

Author here. Happy to answer any questions.

Can you give morespecifics about what you were running on and what you purchased for your own gear?

I run an environment that scales to around 1,000 EC2 instances daily. Primarily we run C3.2Xlarge and R3.2xlarge for the core of our application.

We have ~12 nodes in our mongo cluster, and havent had a single issue with these nodes.

I occasionally get a zombie (totally hung VM) but thats very infrequent. I was aggressively using spot instances previously, but have switched to all 12-month reservations (We would lose many machines to a spot outage, new machines - more than those on Richess) and the recovery time for our system is 35 minutes (due to the R3 boxes needing to download their in-memory index from other machines) - so our service is degraded in capacity until the relaunch of these machines completes.

[aside: if youre looking to use spot, do two things - over-provision by a factor of 1.8 and spread across zones, and go look into using ClusterK.com for their balancer product]

Anyway, Just curious what was causing "sometimes daily" outages - I can't imagine that this would be due to AWS and not lacking ability of your application to handle instance losses.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#33

Post by bobofettfett »

My 5cent:

If you have lots and lots of money and a high margin business, do yourself a favor and go with Amazon (much less hassle with contract management and low level challenges).

If you need to scale month to month and are growing 50% per month, go with Amazon.

If you are very small and can live with 10 instances, go with Amazon.

If CAPEX doesn't help you and for whatever reasons you need to spend OPEX, go with Amazon.

If you need many (types of) machines for failover but which otherwise mostly idle, go with Amazon.

Otherwise it's always cheaper to buy or rent hardware. Amazon is very expensive (TCO).

If you base your decision on hype, you're screwed.

* Amazon stands for Cloud Provider, personally I'm choosing Digital Ocean with Mesos/Docker.

* Except S3 which is a no brainer to use.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#34

Post by TheMagicHorsey »

toomuchtodo wrote: post #12
mattbeckman wrote: post #5

"With Amazon we experienced networking issues, hanging VM instances, unpredictable performance degradation (probably due to noisy neighbors sharing our hardware, but there was no way to know) and numerous other problems. " Why do I get the feeling it was kind of a cop-out to just pack up and move without finding the root cause? I've seen it plenty of times: the "best solution" is to just find a different hosting prov…

> In my experience, I've never found an issue with an application on AWS that wasn't caused by either a misunderstanding of what was being offered (e.g. not provisioning enough PIOPS for database volumes), or simply issues with the application code. You haven't been using Amazon long enough then. Amazon is great for proof of concept. No upfront costs, extremely scalable, etc. Unfortunately, its expensive compared to…

But doesn't Netflix successfully use AWS now?

Its possible they get special treatment if they are big enough (nobody else's jobs on their physical machines ... or something like that).

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#35

Post by jewel »

nanoGeek wrote: post #10

One question. If real hardware was always 50% cheaper then AWS wouldn't have been such a success. Can you please explain in which scenarios it makes economical sense to use AWS compared to real hardware?

Really you'd want to compare EC2 and real hardware, not AWS and real hardware. AWS as a total package comes with a great many services, and if you're using more than a few of them then it can require a great deal of engineering time to set up replacements.

A lot of AWS services can be used by real hardware though, so it's not all or nothing.

For example, where I work we use S3 to store an archive of files but keep the working set of data cached on our web servers which are at codero.

We have video rendering servers which turned out to be much cheaper to do with a cluster of desktop-class hardware in a server closet at our office as opposed to the server grade GPU instances on EC2. The monthly cost of a single GPU instance at EC2 is more than the total cost of the hardware off of newegg.

However, for outages we have a script that spins up GPU instances on EC2 which is much more economical than having a separate set of servers somewhere just in case.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#37

Post by gdulli »

nanoGeek wrote: post #10

One question. If real hardware was always 50% cheaper then AWS wouldn't have been such a success. Can you please explain in which scenarios it makes economical sense to use AWS compared to real hardware?

That's not true. AWS is known to be more expensive in the long run. AWS was never about being cheaper for a mature company. It's popular because it's cheap/easy to get started, it's elastic, and it allows CTOs the freedom to balance capex vs. opex.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#38

Post by sz4kerto »

gaadd33 wrote: post #22
gtrubetskoy wrote: post #20

This is a no-brainer if you've ever done anything at scale. The explanation is rather simple - hardware is always "on the premise", yours or Amazon's. Someone needs to swap drives, motherboards, man the networking gear, run cables, etc. Amazon doesn't really get a break on the hardware cost because 10,000 servers do not cost less per server than 100 servers (in fact they cost more as the volume goes up if you need th…

Why would 10,000 servers cost more than 100 servers? It seems like if you are buying all of your parts in bulk they are going to be cheaper. I'm pretty sure Intel's pricing on CPUs are cheaper by the tray than individually. I know at least when I've bought 20-30 servers at a time, I was able to get a lower cost than when I've only been buying one.

Prices go down until to a certain amount, then then start to increase. Analogy: you want to buy shares of company X. If you buy 1, brokerage costs are high compared to your investment. If you buy 100, you still get the shares from the top of the order book, and fees become negligible. Buy 10M, and you will pay much-much more per share because supply is not going to be there.

Just think of the simple supply-demand curve. As demand increases, price increases as well. The bulk discount pricing is only valid for amounts that provide better utilization of the supply chain. If Intel can produce 1M chips a month, then if somebody orders the last 50k, he might get a discount. If someone wants 2M, then he needs to pay a huge markup because the supply chain is not ready.

And Amazon is definitely big enough to move the equilibrium price up.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#39

Post by gpapilion »

makmanalp wrote: post #27
gtrubetskoy wrote: post #20

This is a no-brainer if you've ever done anything at scale. The explanation is rather simple - hardware is always "on the premise", yours or Amazon's. Someone needs to swap drives, motherboards, man the networking gear, run cables, etc. Amazon doesn't really get a break on the hardware cost because 10,000 servers do not cost less per server than 100 servers (in fact they cost more as the volume goes up if you need th…

> The explanation is rather simple - hardware is always "on the premise", yours or Amazon's. Someone needs to swap drives, motherboards, man the networking gear, run cables, etc. > So you're paying Amazon to do the same work you would do otherwise - only you're subject to their rules and procedures and Amazon being a profitable business needs to mark their services up. But I thought that they were paying Softlayer to…

Generally I'd say the extra cost savings come from the lack of software needed to support the amazon style apis. They may have also made a multi year commitment, which would also further drive the cost down.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#40

Post by Thaxll »

TheMagicHorsey wrote: post #34
toomuchtodo wrote: post #12

Earlier quoted context omitted.

> In my experience, I've never found an issue with an application on AWS that wasn't caused by either a misunderstanding of what was being offered (e.g. not provisioning enough PIOPS for database volumes), or simply issues with the application code. You haven't been using Amazon long enough then. Amazon is great for proof of concept. No upfront costs, extremely scalable, etc. Unfortunately, its expensive compared to…

But doesn't Netflix successfully use AWS now? Its possible they get special treatment if they are big enough (nobody else's jobs on their physical machines ... or something like that).

Netflix uses thousands of instances, they don't share servers.
Post reply on HN