Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: AWS or dedicated server?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 43 posts

Re: Ask HN: AWS or dedicated server?

#31

Post by jeremyw »

I built a 140-machine farm at Softlayer, ~6 months ago. Here are some observations (that may not mean much south of 10 boxes.)

a) AWS feature advantages (mostly instascale, in our case) fade with the high cost of every additional dedicated box.

b) It's nice to virtualize the map of services to boxen, but at some level of scale, each box has a single task and you want the ability to run it _flat out_. If so, you have to decide if Xen overhead is worth labor somewhere else, and alternately, if Xen source compatibility holds you back from new kernel features. (Pick your VM technology.)

c) We still wanted instascale with our own software distribution, so in something less than a week I hand-tooled a pxe-based provisioner that initialized from a live exemplar (gentoo, whee). It took some work to find the right propeller heads at Softlayer, but eventually we understood each other and the bootp listeners got turned off for our subnets. "Insta" became 2-hour hardware activation, which was ok for us. You might consider puppet in the same way (except for Gentoo's long from-scratch build time.)

d) Virtually all provider admin is automated and this still works and makes sense at scale. The SOAP API that backs it is not quite fully baked, but is very useful. Paired with box-level IPMI pokes, you can replicate AWS control over hardware.

e) Whatever AWS provides, at scale you still have a custom setup at some level of abstraction, so putting in place exactly the right hardware saves labor.

f) Substantial discounts can be had.

Re: Ask HN: AWS or dedicated server?

#32

Post by astine »

mdasen wrote: post #7

AWS, no hesitation. AWS isn't that hard to configure. ElasticFox puts a nice GUI to it and while it will take a short while to get used to the AWS way of doing things, you're better off. With AWS, you have a nice spray files everywhere storage in S3, EC2 provides lots of RAM and CPU muscle, EBS provides RAID-level reliable persistent storage for EC2 that can be backed up to multiple data centers with a single API cal…

I find EC2 to be a bit pricey for webapps if thats what we're going for here. It seems to be optimized for high performance and high availability compute clusters, not for persistent webapps. A traditional VPS (like Slicehost) is a more economical solution for that, IMHO.

Re: Ask HN: AWS or dedicated server?

#33

Post by davidw »

raghus wrote: post #19
mdasen wrote: post #7

AWS, no hesitation. AWS isn't that hard to configure. ElasticFox puts a nice GUI to it and while it will take a short while to get used to the AWS way of doing things, you're better off. With AWS, you have a nice spray files everywhere storage in S3, EC2 provides lots of RAM and CPU muscle, EBS provides RAID-level reliable persistent storage for EC2 that can be backed up to multiple data centers with a single API cal…

I use and recommend Slicehost. I'll tweak just one thing and that is that SH offers backups only for slices upto 2GB - for the 4/8/15G slices, there is no backup option. I am not sure why though.

All other things being equal (which I don't know for a fact), Linode is significantly cheaper than Slicehost, considering the differences in memory usage between x86 (32 bit) and x86_64:

http://journal.dedasys.com/2008/11/24/slicehost-vs-linode

Re: Ask HN: AWS or dedicated server?

#34

Post by timf »

shimon wrote: post #23
tdavis wrote: post #16

Earlier quoted context omitted.

Sure (I really should do a real write-up on this...): So the main purposes are simple horizontal scaling and efficient use of hardware. Virtualization makes horizontal scaling simple because it's just a matter of cloning a particular machine (or machines). It makes efficient use of hardware because there's no need to have a dozen physical machines for a dozen different purposes, unless they all use a full machine wor…

Thanks! That's a pretty thorough reply, and I have to agree that it's hard to imagine an easier way to deploy new server instances than copying a VM. I think you can get pretty darn close with a package system like DEB or RPM, but in terms of being able to bring up and test an environment on a development machine, and then be assured the same environment will work exactly the same in production, this is a reassuring…

It's basically hand-made custom EC2, right? Is there an open source project somewhere to package up the tools to do this on one's own servers?

There's two that I know of that support the EC2 wire protocol: Nimbus and Eucalyptus.

I am the primary developer of Nimbus. We had an EC2-like system released before EC2 existed, only later adapting to their protocol because our users wanted to use the EC2 client tools.

Re: Ask HN: AWS or dedicated server?

#35

Post by aristus »

mdasen wrote: post #12
aristus wrote: post #2

I've done AWS, dedicated, and colo. Each one has their own tradeoffs. AWS is daunting at first -- but then so is Debian. Once you figure out the keys thing and your base image it's fairly easy (also see ElasticFox, S3Browser). You might as well learn it, even if you stick with dedicated hosting for other reasons. Things not to sneeze at: * elastic: start up a dozen servers in a few minutes * free access to s3 storage…

So, just to comment on the disk reads: the local, non-persistent disks don't offer great speeds (you're right). However, that shouldn't matter too much now. All of your application code should fit in memory once it's been read off the disk and so you shouldn't be hitting the local disks much after boot. Static files should be on S3 and databases should be stored on EBS. EBS has great performance. RightScale noted tha…

I'll have to try it out, thanks. A quick search doesn't turn up any bonnie tests or similar, so I'll do them and put them online.

(edit) this suggests that EBS is slower than the virtual disk for open/seek/write/flush: http://bizo-dev.blogspot.com/2008/11/disk-write-performance-...

Re: Ask HN: AWS or dedicated server?

#36

Post by timf »

tdavis wrote: post #28
shimon wrote: post #23

Earlier quoted context omitted.

Thanks! That's a pretty thorough reply, and I have to agree that it's hard to imagine an easier way to deploy new server instances than copying a VM. I think you can get pretty darn close with a package system like DEB or RPM, but in terms of being able to bring up and test an environment on a development machine, and then be assured the same environment will work exactly the same in production, this is a reassuring…

Yeah, we should surpass it. As you said though, it's also very much about simplification (following the initial learning, anyway). EC2 may have simplified a little more eventually, but it was also even more expensive and difficult for me to learn up front. As for a custom EC2, it's really the other way around; EC2 is a custom Xen setup. Having initially learned of virtualization before AWS and such cloud services, EC…

"EC2 is the strange one to me. I'd only ever known "DYI EC2", so to speak."

Those EC2-like systems do have their place. There is a lot more to do and think about when you are allowing others to run VMs on your infrastructure. That is often not the case and not your situation either, it sounds like.

Re: Ask HN: AWS or dedicated server?

#37

Post by aquaphile »

AWS -- we run our entire insurance company, and its multiple applications, using the EC2, EBS, S3, SQS, and FPS services provided by AWS. I think the only AWS we haven't used to date is Mechanical Turk. We highly recommend AWS: we started with dedicated hardware years ago, migrated to an excellent virtual host, and then finally moved to AWS last year when they implemented EBS and provided the SLA. Once you have multiple servers, it makes increasing financial sense to use AWS.

Re: Ask HN: AWS or dedicated server?

#38

Post by mdasen »

astine wrote: post #32
mdasen wrote: post #7

AWS, no hesitation. AWS isn't that hard to configure. ElasticFox puts a nice GUI to it and while it will take a short while to get used to the AWS way of doing things, you're better off. With AWS, you have a nice spray files everywhere storage in S3, EC2 provides lots of RAM and CPU muscle, EBS provides RAID-level reliable persistent storage for EC2 that can be backed up to multiple data centers with a single API cal…

I find EC2 to be a bit pricey for webapps if thats what we're going for here. It seems to be optimized for high performance and high availability compute clusters, not for persistent webapps. A traditional VPS (like Slicehost) is a more economical solution for that, IMHO.

Slicehost can be cheaper because of the included bandwidth, but AWS can be really helpful too.

Here's two examples:

Database server. EBS is a high-performance disk system offering speeds of 70MB/s in the real world. Databases love fast disks. Slicehost's disks are slow (at least for high performance databases). Xen doesn't have great disk access speed and they're just plain old local disks. So, if you need to scale up a database, AWS is going to accommodate that better.

Infinidisk! With Slicehost, you have a fixed disk size and you can't buy more. With EBS, you can just keep getting more and more disk - you can even stripe volumes so that they show up as a single disk to your instance.

Life is about trade-offs. I love Slicehost and use them for all my stuff. I've set up EC2 stuff for other people who wanted it. So, it really depends on what you need.

Re: Ask HN: AWS or dedicated server?

#39

Post by lsc »

I would say "do both" - aws is awesome if your site is running slow 'cause you are out of capacity, or you otherwise need a box 'right now' or for only a short period of time. spin up another instance and be done with it. But for the boxes you leave on all the time, you are probably better off buying and co-locating your own server. Usually the capital cost difference is made up in only a few months.

The times when a Xen host makes long-term sense are when you want a box that is smaller than optimal. Right now, I buy dual quad-core opterons w/ 32G ram and 2x1TB disk... assuming I am ok with moderate-speed low-power opterons, it costs about $3K up front. Hosting, say, another $150/month. That's a whole lot of ec2 instances. At those prices, well, AWS is pretty expensive over the long term.

But yeah. AWS is awesome for the servers you don't need on all the time, or servers you don't have time to setup (or your whole ball of wax if your margins are such that paying more for computers won't break your business model.)

Re: Ask HN: AWS or dedicated server?

#40

Post by lsc »

mdasen wrote: post #7

AWS, no hesitation. AWS isn't that hard to configure. ElasticFox puts a nice GUI to it and while it will take a short while to get used to the AWS way of doing things, you're better off. With AWS, you have a nice spray files everywhere storage in S3, EC2 provides lots of RAM and CPU muscle, EBS provides RAID-level reliable persistent storage for EC2 that can be backed up to multiple data centers with a single API cal…

the upside of dedicated servers (that you own) is that after a few months they are dramatically cheaper. Personally, I think that if you are small enough to need less than a full physical box, a VPS usually makes sense. If you are large enough to need more than one physical box, it usually makes sense to use ec2 or other VPS providers for the stuff you need right now, and then build out dedicated servers for your longer-term needs.
Post reply on HN