Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How and Why Swiftype Moved from EC2 to Real Hardware

highscalability.com

Post reply on HN
Print view View original

171–180 of 194 posts

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#171

Post by count »

kovyrin wrote: post #145
risaacs99 wrote: post #141

Earlier quoted context omitted.

Did you reserve your instances? Were you using current generation instances (c3, m3, etc). Did you try to take advantage of traffic patterns to scale up and down the number of instances you were running?

We had reserved instanced and regular ones, we did not see any patterns in stability issues between those. Re: instance types - I do not really remember which instances we were using to be honest. And as for the scaling up and down - we had a hard time keeping it all up as it was, we did not want to spend resources trying to make it work with constantly changing node pools (though I understand, that it would push us…

Did you try dedicated instances?

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#172

Post by geoka9 »

voltagex_ wrote: post #150
toong wrote: post #146

Earlier quoted context omitted.

Can you elaborate why not ? You can get some cheap dedicated hosting (ex: [1]) for a fraction of the price. It's so cheap compared to AWS you can order a few spare ones and still come out cheaper than your one beefy AWS instance ? The only way it doesn't make sense is, if you need to scale up and down very fast ? [1] 60 euro/month: Quad-Core Haswell, 32 GB (non-ECC) RAM, 240Gb SSD @ https://www.hetzner.de

Hetzner is fantastic but if you're not in Europe, what's your latency like?

There's OVH that has a big data center in Montreal.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#173

Post by geoka9 »

Negitivefrags wrote: post #138
babo wrote: post #48

Earlier quoted context omitted.

I would like to know the cost calculation after a year or two. With a handful of servers it's easy to get the false impression that HW failures are rare.

We have 100 physical servers and hardware failures really are very rare. Very rare. In fact over 4 years we have only had 3 hard drives fail and no other hardware failures.

Do you have any plans on replacing your hard drives as they get old? Or you just wait for them to fail?

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#174

Post by jonpress »

I think cloud providers are mostly good for small/medium, fast growing startups. Big companies need more granular flexibility. The path to vertical integration in the software indsutry is a lot smoother than in many other industries, there are many levels:

You can...

1. Make your own hardware

2. Own/manage your own hardware

3. Rent commodity hardware from a standard hosting provider

4. Use IaaS (e.g. EC2)

5. Use PaaS (e.g. AppFog, Nodejitsu)

6. Use BaaS (e.g. Firebase, PubNub, Pusher.com)

The higher the level, the more technical flexibility you lose. The bigger the company, the more it makes sense operate at a lower level because there is no significant wastage being introduced as you move down the levels (you remove the middlemen so you can pocket their profits) and the capital cost to move between levels is relatively low.

If you compare software to another industry like cheesemaking for example, if you're a cheesemaker and you want to make your own milk, the next step is to buy the whole farm and then you have to figure out what do do with the meat (wastage). Going between these two levels is expensive and could mean doubling or tripling your expenses so it's not an easy move to make.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#175

Post by berkay »

toong wrote: post #146
emodendroket wrote: post #136

Earlier quoted context omitted.

If you're a small org with relatively small volume self-hosting doesn't make sense.

Can you elaborate why not ? You can get some cheap dedicated hosting (ex: [1]) for a fraction of the price. It's so cheap compared to AWS you can order a few spare ones and still come out cheaper than your one beefy AWS instance ? The only way it doesn't make sense is, if you need to scale up and down very fast ? [1] 60 euro/month: Quad-Core Haswell, 32 GB (non-ECC) RAM, 240Gb SSD @ https://www.hetzner.de

The cost of the systems is almost irrelevant. The time of qualified people is far more expensive. Sot it's not just the elasticity. You have to look everything from the time perspective as well. What additional knowledge will I need? Will I need to learn load balancers, how to make them highly available? will I have to learn about SSL certificates and termination? Will I need to learn how to implement and operate a secure and highly available DNS service? How much is your time worth? Our hosting costs are a fraction of a single worker's salary. Whether AWS is more expensive is essentially irrelevant. OP argues that it was not doing the job for them which is an entirely different issue.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#176

Post by johnnycarcin »

kovyrin wrote: post #140
johnnycarcin wrote: post #71

Interesting article, thanks for sharing. We actually just went the exact opposite because of the larger scale issues we were having with Softlayer. Do you feel like you lost any resiliency by making the switch to physical servers (more virtual instances on one physical server, servers in the same rack, etc)?

No, I really do not think going to EC2 could be beneficial in any way in terms of improving resiliency compared to Softlayer. SL allows you to control which VLANs your box will end up on. VLANs could be treated as racks (since they do not allocate more than one VLAN per rack). Then you have multiple DCs in one region (e.g. DAL01, DAL05, DAL07, etc) and you have many different regions (DAL, SEA, WAS, AMS, etc). I'd be…

We were about 75% virtual with SL and 25% bare metal. One of our issues with the virtual stuff is when we started dedicating them to a set VLAN, multiple times we ran into an issue where some type of resource for that pod the VLAN was in would be maxed out (usually storage) so we couldn't create a new instance. The solution we were given was to let the system pick a VLAN but by doing that we had lost control of the placement and added some complexity to our architecture.

Aside from that it was mainly nit-picky type stuff, but still things that were annoying (networking issues between DCs, networking issues between pods, internal mirrored apt-get repos going out of sync, API is kind of blah, etc).

We use docker so having a few bare metal machines with tons of containers on them wasn't a great HA setup (for us at least), even running in two data centers. The fairly quick setup time though was a nice selling point.

When we went to AWS things just kind of worked. The API was easier to use and the GUI portal was way nicer/stable. So far we have not had any odd issues with our instances, but we also typically run them at about 50% capacity so that might be why. It is also still early so maybe things will come up in 6+ months that send us back to SL :)

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#178

Post by bnolsen »

digi_owl wrote: post #126
rodgerd wrote: post #61

Earlier quoted context omitted.

> There's not much need for a fancy article on a fancy website in order to understand a key concept of cloud computing: I wish it were true, but plenty of companies are gripped by cloud fever. I've seen quite a few going down the route of charging into the cloud not because they've run the numbers and found it stacks up, but because they want to be in the cloud, and Amazon have some great marketing people.

Sounds akin to the outsourcing binge that went on a decade (?!) or so back. Back then it seemed most companies did it not because they had done the numbers, but because a few big names had done it and so the others did it to apparently piggy back on the stock markets attention.

in fact it is related to outsourcing. in the past outsourcing processing involved shipping drives of data overseas. The risk and schedule hit were big, the offshore companies had crap hardware that usually cost a lot more than it cost in the US. That quieted down for a while. The new cloud stuff got these people interested again in processing outsourcing, only now the data is shipped to google.amazon/whoever and the outsource employees use the cloud vms to run the processing. I wonder how long before this mini trend starts slowing down when they finally figure out that management overhead and poor communication is in most cases the main problem with outsourcing.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#179

Post by [deleted] »

digi_owl wrote: post #126
rodgerd wrote: post #61

Earlier quoted context omitted.

> There's not much need for a fancy article on a fancy website in order to understand a key concept of cloud computing: I wish it were true, but plenty of companies are gripped by cloud fever. I've seen quite a few going down the route of charging into the cloud not because they've run the numbers and found it stacks up, but because they want to be in the cloud, and Amazon have some great marketing people.

Sounds akin to the outsourcing binge that went on a decade (?!) or so back. Back then it seemed most companies did it not because they had done the numbers, but because a few big names had done it and so the others did it to apparently piggy back on the stock markets attention.

[deleted]

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#180

Post by dmourati »

AndyNemmity wrote: post #129
rodgerd wrote: post #61

Earlier quoted context omitted.

> There's not much need for a fancy article on a fancy website in order to understand a key concept of cloud computing: I wish it were true, but plenty of companies are gripped by cloud fever. I've seen quite a few going down the route of charging into the cloud not because they've run the numbers and found it stacks up, but because they want to be in the cloud, and Amazon have some great marketing people.

Often times companies move because their organization doesn't deliver, and the hope is a cloud company will do a better job of it. If you have a great team, I firmly believe hosting yourself is far, far, far less expensive. If you have a terrible team, then cloud (hosting) is less expensive. Even if it was exactly the same cost, you're gaining by not having to have a staff to run it, and the costs of managing them, e…

Reset button is worth a ton in itself. This is a really nice way of looking at it.
Post reply on HN