Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How and Why Swiftype Moved from EC2 to Real Hardware

highscalability.com

Post reply on HN
Print view View original

161–170 of 194 posts

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#161

Post by risaacs99 »

kovyrin wrote: post #145
risaacs99 wrote: post #141

Earlier quoted context omitted.

Did you reserve your instances? Were you using current generation instances (c3, m3, etc). Did you try to take advantage of traffic patterns to scale up and down the number of instances you were running?

We had reserved instanced and regular ones, we did not see any patterns in stability issues between those. Re: instance types - I do not really remember which instances we were using to be honest. And as for the scaling up and down - we had a hard time keeping it all up as it was, we did not want to spend resources trying to make it work with constantly changing node pools (though I understand, that it would push us…

Well I ask because reserved instances can significantly reduce the price of ec2 (up to something like 75%). Also just turning off idle instances can save a ton. If you invest the time in doing those things I think you can beat other options and so in that way, cloud infrastructure can be very economical.

It is like many other things involved in running a technology company. Investing in automation can pay off hugely.

The newer instance types are very reliable too (in my experience).

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#162

Post by snewman »

Negitivefrags wrote: post #132
lucas_alwin wrote: post #87

Here's my anecdotal, one-data-point experience from moving a giant EC2 environment to datacenter: 1. Your operational overhead will increase _a lot_. Be ready to hire on a lot of ops staff if you expect them to do anything but put out fires. And as you grow you'll need experts, people like network engineers. 2. Any weirdness you experienced with AWS infrastructure will be replaced with weirdness in your own environme…

Contrast that to my experience with renting from Softlayer: 1) The operational overhead is the same as Amazon. Actually I think it's less because everything is so predictable. Every machine we get has identical performance to any other. We still don't have to care about the exact same list of things that EC2 provided but we also don't have to care about weird cloud issues. 2) See above answer but also Softlayer are h…

Can you walk me through a concrete example of cost savings? Every time I look at something like SoftLayer, it doesn't seem to offer much of a savings compared to a corresponding EC2 reserved instance. Sure, you can do way better than EC2 on-demand instances, but the reasonable comparison is to a reserved instance.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#163

Post by kovyrin »

guru_meditation wrote: post #160
snoopybbt wrote: post #43

There's not much need for a fancy article on a fancy website in order to understand a key concept of cloud computing: Cloud computing offers you the great and awesome advantages of being able to instantly scale your application, replicate your data and basically just grow according to your business volume, and all this without significant investments, delivery time, setup time, people time, maintenance or anything bu…

The reason why it is extremely hard to engineer robust large scale AWS cloud apps can be summarized under the umbrella of performance variance : - machine latency varies more, you can't control it - network latency varies more - storage latency varies more (S3, Redshift, etc.) - machine outages are more frequent where more can be an order of magnitude more variation than on bare metal deployments. I am not saying the…

Great comment! Thanks a lot for sharing this.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#164

Post by serve_yay »

gtrubetskoy wrote: post #20

This is a no-brainer if you've ever done anything at scale. The explanation is rather simple - hardware is always "on the premise", yours or Amazon's. Someone needs to swap drives, motherboards, man the networking gear, run cables, etc. Amazon doesn't really get a break on the hardware cost because 10,000 servers do not cost less per server than 100 servers (in fact they cost more as the volume goes up if you need th…

Yeah, and you pay McDonald's to cook the burger for you -- but you can't do it cheaper than they can.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#165

Post by gtaylor »

snoopybbt wrote: post #43

There's not much need for a fancy article on a fancy website in order to understand a key concept of cloud computing: Cloud computing offers you the great and awesome advantages of being able to instantly scale your application, replicate your data and basically just grow according to your business volume, and all this without significant investments, delivery time, setup time, people time, maintenance or anything bu…

I see there are already some contrarian-because-I-feel-like it replies, but this is a really succinct way to put it. Of course there are exceptions and edge cases, but EC2 is all about elasticity. If that doesn't appeal to you, don't touch EC2. If you aren't using the "E" in EC2, there are far cheaper, more performant alternatives.

On a side note, I feel like EC2 is simultaneously the best and worst service on AWS.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#166

Post by corford »

merb wrote: post #105
chx wrote: post #89

Earlier quoted context omitted.

> Your operational overhead will increase _a lot_. Be ready to hire on a lot of ops staff if you expect them to do anything but put out fires. And as you grow you'll need experts, people like network engineers. Why? What are you talking about? You are hiring servers you are no colo'ing them. Networking them is not your problem. Your responsibility still starts from a root prompt just there's no VM layer between that…

and if you have more than one? or more than one dc? somebody needs to connect it, or you will need a VPN. it's not that easy without Clouds if you need connected servers. We switched to AWS since connecting servers in a dc isn't as cheap as people think of.

I host all my stuff with Hetzner (who are at the cheap end of the market compared to Softlayer) and even they provide pre-configured VLANS and private switches. For intra-DC you can just setup some redundant openvpn links or pay the dc to configure a hardware tunnel between both sites.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#167

Post by corford »

kovyrin wrote: post #118
arjunnarayan wrote: post #47

Earlier quoted context omitted.

Could you please list some competitors to SoftLayer? It's hard to get reliable opinions on cloud providers backed up with actual experience. I'd really appreciate it!

My personal experience within the last 7 years: * Softlayer – the best option in terms of quality of service, quality of hardware, quality and size of the infrastructure (geo distribution, etc). * Rackspace – nice, until you grow enough to get relatively locked in and then your prices start to go up, provisioning time suffers and their service turns into shit. * Steadfast – provisioning times up to a week, basically…

Re: Hetzner - they do offer some cheap "desktop" grade servers but they also offer lots of real servers: https://www.hetzner.de/ot/hosting/produktmatrix/rootserver-p... (e.g. 120GB RAM, Xeon chips, SAS drives, hardware raid etc.)

I've been running three 32GB servers (each with 3TB storage) with them for 2+ years now and the only outage I've experienced is the switch (5 port GBit) dying once. Hetzner tech replaced it in under an hour.

These three servers cost me €263/month (that's total, not each). Included in that monthly price is an additional IPv4 for each server, a private 5port Gbit switch, remote console access and 300GB of DC backup space.

There are probably better deals available now (i.e. more RAM at the same price) than the one I'm on since it's old and not offered on their site any more (/makes note to self to call Hetzner sales)

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#168

Post by corford »

davidjgraph wrote: post #26

It strikes me that more and more a critical selection when growing in this way is the DNS part. It needs to be back-end agnostic and provide an increasing amount of functionality. Health checks and failover are must have now, but this article makes me wonder three things: 1) Are there any DNS services that understand geography of your "zones", i.e. route to and failover based on IP? (but are still platform agnostic).…

Check out DynECT from http://www.dyn.com

Twitter, Mozilla and lots of other big names use them. I remember watching a webcast where Mozilla said they used Dyn's anycast failover service, with TTLs on their domains set to 5 seconds.

I've been using their DynECT entry level package ($30/month) for a couple of years and it's great.

Edit: you might also find this comment from an old thread interesting/useful: https://news.ycombinator.com/item?id=7813589 (go up two levels to phil21's first comment - HN isn't giving me a direct link sadly)

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#169

Post by rodgerd »

twblalock wrote: post #112
lucas_alwin wrote: post #87

Here's my anecdotal, one-data-point experience from moving a giant EC2 environment to datacenter: 1. Your operational overhead will increase _a lot_. Be ready to hire on a lot of ops staff if you expect them to do anything but put out fires. And as you grow you'll need experts, people like network engineers. 2. Any weirdness you experienced with AWS infrastructure will be replaced with weirdness in your own environme…

Operation staff will immediately start guarding the food bowl as resources become finite. Server provision waits start to seem like breadlines. Power is consolidated with Those Whom You Must Ask. This is the worst part about moving out of the cloud, especially since cloud computing has moved a lot of ops and deployment responsibility to developers.

I would say an absolute pre-req for moving from Amazon to own servers would be tooling like VCAC or OpenShift to give people a nice self-service experience.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#170

Post by wmf »

rodgerd wrote: post #169
twblalock wrote: post #112

Earlier quoted context omitted.

Operation staff will immediately start guarding the food bowl as resources become finite. Server provision waits start to seem like breadlines. Power is consolidated with Those Whom You Must Ask. This is the worst part about moving out of the cloud, especially since cloud computing has moved a lot of ops and deployment responsibility to developers.

I would say an absolute pre-req for moving from Amazon to own servers would be tooling like VCAC or OpenShift to give people a nice self-service experience.

Then you get "out of capacity" errors even sooner. You could have a pretty bad experience with an internal cloud that was self-service but wasn't run like a service on the business side, so the fixed annual budget was exhausted immediately.
Post reply on HN