Live data from Hacker News

How and Why Swiftype Moved from EC2 to Real Hardware

highscalability.com

111–120 of 194 posts

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#111

Earlier quoted context omitted.

Hardware is almost always cheaper than engineering time. Don't optimize prematurely.

Yes, but engineering is cheaper than running after real-time problems.

Now we get to the technical debt debate. Sometimes, you have to make good decisions now instead of perfect decisions later. The market doesn't care how elegant your code is.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#112

Here's my anecdotal, one-data-point experience from moving a giant EC2 environment to datacenter: 1. Your operational overhead will increase _a lot_. Be ready to hire on a lot of ops staff if you expect them to do anything but put out fires. And as you grow you'll need experts, people like network engineers. 2. Any weirdness you experienced with AWS infrastructure will be replaced with weirdness in your own environme…

Operation staff will immediately start guarding the food bowl as resources become finite. Server provision waits start to seem like breadlines. Power is consolidated with Those Whom You Must Ask.

This is the worst part about moving out of the cloud, especially since cloud computing has moved a lot of ops and deployment responsibility to developers.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#113

One question. If real hardware was always 50% cheaper then AWS wouldn't have been such a success. Can you please explain in which scenarios it makes economical sense to use AWS compared to real hardware?

Well for me personally it makes sense if you are not of very large scale. Real hardware is cheaper but the costs of electric, transit, and colo space are not your only costs. If you are going to do physical hosting right you need to be in several locations, and you need to have a very good idea of what assets you will need for a period of time. With AWS I can scale up easy, not have to worry about doing things like r…

With Softlayer I get multiple locations (transparently routed 100% transparent backend connectivity), get 1-2 hrs provisioning speed, I do not need to worry about networking and hardware. Failed drives (when they fail, which happens rarely since the hardware they use is REALLY good) are replaced within an hour.

Colocation is a very different beast and I certainly would not encourage anybody to do that until a very large scale when rented hardware economics stop working for them.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#114

I'm convinced hybrid cloud is the way to go. Anything needing high IO performance should be on dedicated. Anything needed CPU/memory elasticity (worker nodes, etc) should be in the cloud. Assuming you can get low latency connectivity into AWS with DirectConnect, this might work?

Cpu also generally sucks due to cache thrashing on non-trivial datasets

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#116

"With Amazon we experienced networking issues, hanging VM instances, unpredictable performance degradation (probably due to noisy neighbors sharing our hardware, but there was no way to know) and numerous other problems. " Why do I get the feeling it was kind of a cop-out to just pack up and move without finding the root cause? I've seen it plenty of times: the "best solution" is to just find a different hosting prov…

I've seen and fought issues with hanging and/or stalling EC2 instances, and the decision was made to move to physical hardware - it was one of those tradeoff choices you have to make about paying N dollars to throw money at hardware vs. M dollars at person-hours to investigate. It's definitely kicking the can down the road (eventually you have to build such that failing infrastructure is transparent to your eng team)…

I've had the odd problem with hanging/stalling instances before I learned to design for the cloud. Now those instances automatically get replaced with new ones and nobody would even notice if I didn't pipe the notice through to Slack.

Of course, this only works if whatever you use it for allows for this.

However, in most cases not using EC2 as a stateless throwaway computing resources is simply a matter of bad infrastructure design.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#117
post #78
post #69

Earlier quoted context omitted.

>you're also paying Amazon's profits. Are you? Amazon is not known for posting profits, and is quite possibly operating AWS at a loss. They have promised to show financial results for AWS itself with the quarterly report being released next month, so we'll see.

Profit for something like AWS is likely difficult to put a number on. They built it so they could run Amazon.com off the infrastructure, but then decided they could make money leasing their under-subscribed portions. If they are not making a cut-and-dry profit from AWS (I'd fathom they are, they are one of the most expensive cloud providers and many other providers turn a healthy profit), then they are at least drama…

Actually, they built AWS as a business on its own right from the start and only later moved Amazon.com onto AWS.

http://www.networkworld.com/article/2891297/cloud-computing/...

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#118
post #24

Earlier quoted context omitted.

No, there is significantly more complexity, overhead, and R&D to providing cloud services in comparison to bare metal. SoftLayer is actually a very expensive bare metal server provider. There are several good options that cost less than 1/3rd the price. Realistically, at just modest scale (a few physical servers), you should see 1/6th the cost of Amazon. The main benefits of Amazon is that it: a) allows you to scale…

Could you please list some competitors to SoftLayer? It's hard to get reliable opinions on cloud providers backed up with actual experience. I'd really appreciate it!

My personal experience within the last 7 years: * Softlayer – the best option in terms of quality of service, quality of hardware, quality and size of the infrastructure (geo distribution, etc). * Rackspace – nice, until you grow enough to get relatively locked in and then your prices start to go up, provisioning time suffers and their service turns into shit. * Steadfast – provisioning times up to a week, basically a joke in today's world. * Some German/EU providers like Hetzner – dirt cheap option with desktop-like hardware, failing quickly. Service is nowhere near SL level.

I could go on and on about those, but other options were even more painful.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#119

Earlier quoted context omitted.

Yes, but engineering is cheaper than running after real-time problems.

Now we get to the technical debt debate. Sometimes, you have to make good decisions now instead of perfect decisions later. The market doesn't care how elegant your code is.

The market cares about uptime, and cost relative to return.

Perfect should never be the enemy of the good.

Re: How and Why Swiftype Moved from EC2 to Real Hardware

#120

The biggest problem with AWS is the outrageous cost of bandwidth. Even if you ignore all the other cost differentials, the bandwidth charges will kill you at scale. Unfortunately cloud computing, or at the very least AWS, overpromises and underdelivers at scale. All in all economically viable use cases for cloud computing are very few and very specific at scale.

Since people are sharing their experiences in this thread, has anyone tried using Direct Connect to get cheaper bandwidth?
Post reply on HN