Live data from Hacker News

How and Why Swiftype Moved from EC2 to Real Hardware

highscalability.com

181–190 of 194 posts

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#181

Earlier quoted context omitted.

Did you reserve your instances? Were you using current generation instances (c3, m3, etc). Did you try to take advantage of traffic patterns to scale up and down the number of instances you were running?

We had reserved instanced and regular ones, we did not see any patterns in stability issues between those. Re: instance types - I do not really remember which instances we were using to be honest. And as for the scaling up and down - we had a hard time keeping it all up as it was, we did not want to spend resources trying to make it work with constantly changing node pools (though I understand, that it would push us…

If you were using old generation instances (t1 m1 c1 etc.) your experience would have been VERY different to current generation instances (c3 c4 m3 t2). Did you try network optimized instances with (very) low latency networking?

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#182

Earlier quoted context omitted.

Prices go down until to a certain amount, then then start to increase. Analogy: you want to buy shares of company X. If you buy 1, brokerage costs are high compared to your investment. If you buy 100, you still get the shares from the top of the order book, and fees become negligible. Buy 10M, and you will pay much-much more per share because supply is not going to be there. Just think of the simple supply-demand cur…

I see no explanation for why suppliers wouldn't match demand. Aside from the HDD shortage which hit everyone, I've seen no issues like you're describing where essentially the market runs out of servers/CPUs/etc. Because that's the only way this theory applies, if they're completely unable to meet the demand due to some specific shortage in the market. The only reason your shares analogy makes sense is because there A…

There are a lot of reasons why suppliers might not match demand.

Let's say you have a factory that runs at 90% utilization and somebody crawls out of the woodwork who wants to order 3 factory-months worth of widgets, delivered next week.

Well first of all, you cannot meet that schedule, so you turn away the order in the instant case.

Now the question is: if we were to scale up production, what is the chance that some new person will crawl out of the woodwork with a similar instant order once the factories are ready? Because if we judge what has happened a one-off case, then we will refuse to meet the demand, whether it is real or not.

(Of course we're also making a lot of simplifying assumptions here like that you have access to capital, that there is no regulatory issue with scaling up production, that increased production does not open you to new lines of attack from your competitors, etc. Which are not good assumptions in general.)

It is our judgment of the demand, rather than the real demand, that controls production. If we are manufacturing, say, kevlar vests in 2001, we may very well interpret a large order as representing an underlying demand shift. On the other hand, if our widgets are luxury cars in 2008, we may interpret a similar set of facts as a one-off order.

The insight here is that real demand is not known at the time that supply is trying to meet it; it is estimated. The extent to which the market clears depends on how good the estimation is. With something like oil we understand demand fairly well, but in markets like consumer electronics the demand predictions are poor. That is why on the one hand Apple is chronically short of iPhones and simultaneously Amazon cannot give its phones away: all the estimates were off.

In short, the more your widget is impacted by technological or cultural shocks, the more likely it is that suppliers won't adjust to meet demand.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#183
post #70
post #61

Earlier quoted context omitted.

> There's not much need for a fancy article on a fancy website in order to understand a key concept of cloud computing: I wish it were true, but plenty of companies are gripped by cloud fever. I've seen quite a few going down the route of charging into the cloud not because they've run the numbers and found it stacks up, but because they want to be in the cloud, and Amazon have some great marketing people.

From personal experience this is absolutely true. The initial cost to migrate and learn the cloud practice is unbelievable, but once you have the process in place running on Amazon CAN save some cost down the road, including the hours you need to replace hardware. You will build tools or use existing tools to create your infrastructure and operation process. A generous estimation to reach a level of maturity is 1.5 y…

But if the alternative to reserved instances is buying servers then the paying upfront thing isn't so much of an issue.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#184
post #114

Earlier quoted context omitted.

Cpu also generally sucks due to cache thrashing on non-trivial datasets

Heh. So what is a good use case for EC2?

When you have highly variable load.

eg. Netflix probably spins up thousands of servers for a few hours.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#185

Earlier quoted context omitted.

Contrast that to my experience with renting from Softlayer: 1) The operational overhead is the same as Amazon. Actually I think it's less because everything is so predictable. Every machine we get has identical performance to any other. We still don't have to care about the exact same list of things that EC2 provided but we also don't have to care about weird cloud issues. 2) See above answer but also Softlayer are h…

Can you walk me through a concrete example of cost savings? Every time I look at something like SoftLayer, it doesn't seem to offer much of a savings compared to a corresponding EC2 reserved instance. Sure, you can do way better than EC2 on-demand instances, but the reasonable comparison is to a reserved instance.

I think a big part of it is that dedicated metal and "reserved instances" are not apples-to-apples. You want absolute control, consistency, and customization of the build, you can buy/lease metal and co-locate it, or you can rent it from someone like SoftLayer, and you still don't ever have to walk into a data center.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#186
post #91

Earlier quoted context omitted.

Yeah, it's always fun when I tell someone at we're 100% self-hosted too. They never stop to think that traffic, load, and usage patterns on craigslist are pretty well understood at this point. Plus it's certainly cheaper than paying someone else to run all your hardware. You get to pick your peers, pick your hardware, and allocate as you see fit.

It's cheaper if you have a good team, which it sounds like you do. A ton of places don't have a good team, and the cost numbers end up being a lot closer. A great team will destroy Cloud margins. An Average-Bad team will fall in line, plus you don't have to run the team anymore.

Your comment makes sense, but it implies Netflix does not have a great team, or that it has other considerations than pure financial ones?

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#187

Earlier quoted context omitted.

Contrast that to my experience with renting from Softlayer: 1) The operational overhead is the same as Amazon. Actually I think it's less because everything is so predictable. Every machine we get has identical performance to any other. We still don't have to care about the exact same list of things that EC2 provided but we also don't have to care about weird cloud issues. 2) See above answer but also Softlayer are h…

Can you walk me through a concrete example of cost savings? Every time I look at something like SoftLayer, it doesn't seem to offer much of a savings compared to a corresponding EC2 reserved instance. Sure, you can do way better than EC2 on-demand instances, but the reasonable comparison is to a reserved instance.

Inter-DC network traffic is not rated/charged for SoftLayer. The key cost differentation is network charges when contrasting SoftLayer other services.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#188
post #91

Earlier quoted context omitted.

Yeah, it's always fun when I tell someone at we're 100% self-hosted too. They never stop to think that traffic, load, and usage patterns on craigslist are pretty well understood at this point. Plus it's certainly cheaper than paying someone else to run all your hardware. You get to pick your peers, pick your hardware, and allocate as you see fit.

It's cheaper if you have a good team, which it sounds like you do. A ton of places don't have a good team, and the cost numbers end up being a lot closer. A great team will destroy Cloud margins. An Average-Bad team will fall in line, plus you don't have to run the team anymore.

I've had many clients on AWS rack up six-figure monthly bills unnecessarily simply because they didn't realize how to design for cost.. it's not a silver bullet. Those penny fractions add up fast. There are a lot of tricks to save money.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#189
Cost-wise, AWS makes a lot of sense when growth is not easy to project, happens rapidly, or varies (seasonally, randomly, etc).

Of course, some of this has to do with your team's skill level, but I've had clients run up $100k+ monthly bills at AWS with a relatively small build-out. (and, wow, VPC migrations..)

For fixed or predictable growth patterns on a mature app/platform, a slow build-out on real iron will generally be significantly less expensive, all other things being equal.

However, there are other advantages to AWS that gets lost in this story, such as pre-built, highly scalable datastores. Comparing EC2 to real iron misses most of the real story on why the cloud is changing everything.

One of the hardest things I have to tell clients is not to build their own datastore/database in house or at EC2; sometimes the case is clearcut, and sometimes not so much, but if you have a datastore at AWS that gives you 80% of what you need, use it instead of rolling your own. (source: IAMA AWS Growth Architect)

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#190

There's not much need for a fancy article on a fancy website in order to understand a key concept of cloud computing: Cloud computing offers you the great and awesome advantages of being able to instantly scale your application, replicate your data and basically just grow according to your business volume, and all this without significant investments, delivery time, setup time, people time, maintenance or anything bu…

Not sure what you mean with "datacenter". If you mean using co-location or similar it probably works out as a small price difference. If you start with an empty room or an empty piece of ground it might cost you a lot to start building everything and hire people to manage it all. Remeber that the big cloud providers can get a scale benefit where a lot of virtual computers share a small operations team. I do think that physiccal servers usually are faster than cloud computers meaning you might be able to use fewer of them to compensate for the price difference.
Post reply on HN