Ask HN: Buying servers?

Another HP story. Not nearly on the scale you're talking about, but the company I works for uses HP DL360 G5's as the hardware they sell to customers. Before we ship to customers, we open the box and install software and such. As a tester/engineer for this software, I've never seen a dead DL360 arrive. Yes, the shipping containers are huge. HP's iLO2 software (integrated lights-out) is excellent.

Also, whatever machine you get, make sure you buy some extra rails and hardware. Store the extras somewhere and label them with the date purchased and which systems they are for.

I have very little experience with EC2 (or managing server farms). I have worked at Amazon and been responsible for an internal service there that ran on a server farm, but we didn't use EC2 and another department managed the machines.

Has EC2 been reliable enough lately? I've heard horror stories of 8+ hour downtime multiple times a year.

I like Sun servers, they can be expensive, but you can get some good deals if you are accepted into SSE (Sun Startup Essentials). For me one of the best things about them is the ILOM, which is badass as far as I'm concerned. Also, their N1 System manager lets you manage and provision a bunch of servers from a single location. It will also keep track of the environment on all of them, but thats nothing you couldn't build with SNMP traps etc. Also, I must mention that if you pay for the support, it is awesome, I had a problem with system drives on a Friday, and they overnighted it and I had it Saturday. They also don't charge you unless you don't send the defective part back, with THEIR free packaging and shipping. It was also nice of them to include 3-4 3M anti-static wrist straps with my shipment of new hardware :)

If you don't need to cram a lot of boxes into a tight space then 2U will usually give you better airflow and cooling.

Also, when those on-motherboard NICs die, you can more easily/cheaply add additional NIC cards to the 2U boxes.

In terms of power, that's actually the main limitation in most colo's these days. Some colos are down at 15 amps per rack, many seem to be in the 20 amps per rack, and some you can get up to 30 but you will pay for that one way or another.

Even at 30 amps per rack, for the kind of machines that you're talking about you are still likely to be less than about 20 machines per rack. I just spec'd a build out of the same horsepower & ram but with more juice for e.g., FC cards to a SAN and the safe limit was 16 machines in a 30 amps per rack colo and 8 machines in a 15 amps per rack colo.

Shopping around for power information can be tricky because some of the colos like to play marketing games to get you to think there price is better than others but the power charges can eat you alive if you need more than what they allocate in their base charges.

Lots of great comments so not much add/kinda like others have said:

When dealing direct with vendors (Dell, Sun and IBM from experience):

1) Price you're given is not only based on current purchase, but future purchase. Telling your sales rep 'I need X servers now, but I expect to need a further X per quarter growing to X after X time' then they'll fight harder for your business. Basically, present your business as a growth opportunity to the vendor.

2) When speaking to your first sales rep, ask at what price/order level they generally deal in. The reps who handle bigger clients have access to bigger discounts.

3) Don't under or over-specify your machines without calculating your later options. Buying an extra processor for one machine now will be a hell of a lot cheaper/easier than buying the matching processor in 3/6/12 months. In the same respect, not buying that extra processor now might be more prudent because a new chassis with the latest and greatest processor in 6/12/24 months might be a better option. (should be part of your scalability predictions anyway).

4) Know your stuff.

4.1) We all know reps bs, by knowing what you're talking about you can reduce this tedium.

4.2) If you ask which processor is better, then it'll always be the more expensive one (or more importantly, the one with the better margin for them which is not always the most expensive).

4.3) mmmurf said it, but to re-iterate. Time = Money. If a rep is holding your hand too much, then you won't get as good deals.

5) Know your vendor and their enemies. 5.1) When is there end of quarter? 5.2) How have they done compared to comparable vendors over the last quarter, year, etc? 5.3) Which vendors are pushing X line/type of servers at the moment? 5.4) Have comparison quotes.

6) Specify a date when you will be looking to purchase. 5.1 has a MASSIVE impact on this.