This is somewhat unrelated, but I remember reading in one of Mixpanel's job posts that they had over 200 servers. 200 for a company of their size that charges by the data point seems kind of a lot. I've worked at a couple tech companies who get by with an order of magnitude less servers and deal with the same load that I bet they deal with. So either they were exaggerating by re-defining what a "server" was in the cloud, they have tons of (costly) freeloaders, or their infrastructure is inefficient.
Why We Moved Off The Cloud
31–40 of 154 posts
Re: Why We Moved Off The Cloud
#32Facebook's CTO thinks the opposite or perhaps even alike.
IMHO, a startup should not start with dedicated, but perhaps when you get to a certain size, perhaps, dedicated hardware, team and bandwidth is the way to go.
Just like you will not have a dedicated chef to cook your meal at start, but rather outsource meals, etc.
-- http://www.bbc.co.uk/news/business-12406171
What's the biggest technology mistake you've ever made - either
at work or in your own life?
Prior to Facebook, I was the chief executive of a small internet
startup called FriendFeed.
When we started that company, we were faced with deciding whether
to purchase our own servers, or use one of the many cloud hosting
providers out there like Amazon Web Services.
At the time we chose to purchase our own servers. I think that was
a big mistake in retrospect. The reason for that is despite the
fact it cost much less in terms of dollars spent to purchase our
own, it meant we had to maintain them ourselves, and there were
times where I'd have to wake up in the middle of the night and
drive down to a data centre to fix a problem.Re: Why We Moved Off The Cloud
#33Good writeup. Key part (lessons learned):
"After deciding to go dedicated, the next step is choosing a provider. We got competing quotes from a number of companies. One thing that I was surprised by — and this really doesn’t seem to be the case with the cloud — is that pricing is highly variable and you have to be prepared to negotiate everything. The difference between ordering at face value and either getting a competing quote or simply negotiating down can be as much at 50-75% off. As an engineer, this type of sales process is tiring, but once you have a good feel for what you should be paying and what kind of discount you can reasonably get, the negotiations are pretty quick and painless.
We ultimately decided to go with Softlayer for a number of reasons:
- No contracts. I don’t think I really need to explain the advantage. You would think that you could get better prices by signing 1 or 2 year contracts, but interestingly enough, out of the initial 5 providers we talked to the two that didn’t require contracts had the best prices.
- Wide selection. Softlayer seems to keep machines around for a while and you can get very good deals on last year’s hardware. Most of the other providers we contacted would only provision brand new hardware and you pay a premium.
- Fast deployment. Softlayer isn’t quite at the cloud level for deployment times, but we usually get machines within 2-8 hours or so. That’s good enough for our purposes. On the other hand, a lot other hosting companies have deployment times measured in days or worse.
One last thing about getting dedicated hardware. It’s cheaper… a lot cheaper. We have machines that give us 2-4x performance that cost less than half as much as their cloud equivalents and we’re not even co-locating (which has its own set of hassles)."
Re: Why We Moved Off The Cloud
#34Just about everyone and their mom hopped onto the cloud bandwagon there for good time, but with this steady onslaught of praise for bare-metal hosting, maybe this will reverse a little. It seems like the only thing cloud really provides best is for: 1) Short lived instances or "now" instances. 2) and... what? I'm trying to think of other situations where cloud beats bare metal, but I am coming up short.
Amazon S3 is awesome and different from intrastructure in the cloud. If you choose to use S3 and a lot of your servers rely on fast access to it, then using EC2 too can often make sense.
Rackspace offers a good compromise where they have cloud services like Cloud Files and Cloud Servers, but you can also have dedicated servers with fast access to your cloud stuff.
As others have mentioned, it’s a lot easier to achieve geographical diversity with a cloud provider like AWS.
Another thing to keep in mind with the AWS cloud: if you have a huge setup, you can always start the largest instances, and you will almost certainly have the physical server all for yourself. And for a fee of $10 per hour per region, across _all_ your instances, you can have totally dedicated instances where you’re guaranteed that none of your instances will be on shared hardware.
($7,200 per region per month sounds like a lot, but it is a fixed fee and a drop in the bucket for people with huge EC2 deployments.)
Re: Why We Moved Off The Cloud
#35Facebook's CTO thinks the opposite or perhaps even alike. IMHO, a startup should not start with dedicated, but perhaps when you get to a certain size, perhaps, dedicated hardware, team and bandwidth is the way to go. Just like you will not have a dedicated chef to cook your meal at start, but rather outsource meals, etc. -- http://www.bbc.co.uk/news/business-12406171 What's the biggest technology mistake you've ever…
Softlayer manages our boxes so none of us are waking up and driving down to any datacenters.
For us, you should keep in mind, the big takeaway is consistent performance not just price.
To be honest, I've never had more sleep in the past 2 years ;).
Re: Why We Moved Off The Cloud
#36Facebook's CTO thinks the opposite or perhaps even alike. IMHO, a startup should not start with dedicated, but perhaps when you get to a certain size, perhaps, dedicated hardware, team and bandwidth is the way to go. Just like you will not have a dedicated chef to cook your meal at start, but rather outsource meals, etc. -- http://www.bbc.co.uk/news/business-12406171 What's the biggest technology mistake you've ever…
Mixpanel isn't purchasing their own servers, they are still having a third party rack, cable, and repair their gear.
Re: Why We Moved Off The Cloud
#37This is somewhat unrelated, but I remember reading in one of Mixpanel's job posts that they had over 200 servers. 200 for a company of their size that charges by the data point seems kind of a lot. I've worked at a couple tech companies who get by with an order of magnitude less servers and deal with the same load that I bet they deal with. So either they were exaggerating by re-defining what a "server" was in the cl…
Mixpanel is an analytics company.
Analytics is server-intensive.
Re: Why We Moved Off The Cloud
#38This is somewhat unrelated, but I remember reading in one of Mixpanel's job posts that they had over 200 servers. 200 for a company of their size that charges by the data point seems kind of a lot. I've worked at a couple tech companies who get by with an order of magnitude less servers and deal with the same load that I bet they deal with. So either they were exaggerating by re-defining what a "server" was in the cl…
Or you bet wrong about the load they deal with.
They may also have higher availability requirements than most companies and need 2X (more?) the infrastructure to protect against a data collection failure.
They may be counting nodes used periodically, e.g. a large Hadoop map-reduce run.
Edit: don't get me wrong -- 200 servers is a lot. :)
Re: Why We Moved Off The Cloud
#39> One last thing about getting dedicated hardware. It's cheaper a lot cheaper. We have machines that give us 2-4x performance that cost less than half as much as their cloud equivalents and we're not even co-locating. I've been saying this for ages, and every time people would fall over backwards trying to defend/prove their cloud mistake... "The cloud is cheaper, faster, and infinitely scalable." Except none of thos…
So I'm running operations at Blekko, and just prior to that I was at Google in their eng/ops organization. I had been doing a whole lot of 'total cost of ownership' aka TCO computations around engineering infrastructure both for Google and of course now for Blekko. The conclusion I came to is that for a 'web 2.0' type setup, the break even point was about 500 'machines.' That was in part because a 'machine' today has…
This may be true if the available virtual servers fit your configuration needs. If they don't individually provide enough IO bandwidth, storage space, RAM, CPU time, etc and you have to add a second virtual server to make up for the lacking attribute the ratio could change from 10:1 to 20:1, making dedicated servers an even better option.
The more custom your configuration the better bare metal is.
Re: Why We Moved Off The Cloud
#40> One last thing about getting dedicated hardware. It's cheaper a lot cheaper. We have machines that give us 2-4x performance that cost less than half as much as their cloud equivalents and we're not even co-locating. I've been saying this for ages, and every time people would fall over backwards trying to defend/prove their cloud mistake... "The cloud is cheaper, faster, and infinitely scalable." Except none of thos…
So I'm running operations at Blekko, and just prior to that I was at Google in their eng/ops organization. I had been doing a whole lot of 'total cost of ownership' aka TCO computations around engineering infrastructure both for Google and of course now for Blekko. The conclusion I came to is that for a 'web 2.0' type setup, the break even point was about 500 'machines.' That was in part because a 'machine' today has…
You should write this up and post it somewhere. It sounds like something I'd like to read the details of.