Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack

forbes.com

Post reply on HN
Print view View original

71–80 of 128 posts

Re: Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack

#71

Post by incision »

hp50g wrote: post #62
incision wrote: post #59

Earlier quoted context omitted.

>It's snake oil. As opposed to a miracle elixir that cures all ills? It's a tool. Like most tools, it's best used when you thoroughly understand the challenge and how/why to apply that tool to it. The problems you've noted here seem more indicative of poor planning and understanding of the solution than any intrinsic deficiency of virtualization.

Its a shit tool that creates issues and costs fuckwads of cash. That is it. We understand it. The numbers sold are not the numbers gained. The entire infrastructure deployment was planned around virtualization, DC failover and resilience using VMware's guidelines and solutions (VMware were even paid to consult on this). It never delivered and doubled our administrative and licensing overhead.

>Its a shit tool that creates issues and costs fuckwads of cash. That is it.

Calm down please.

>The entire infrastructure deployment was planned around virtualization, DC failover and resilience using VMware's guidelines and solutions (VMware were even paid to consult on this). It never delivered and doubled our administrative and licensing overhead.

That might say something about VMWare's specific solution and you being taken for a ride, but it's nothing intrinsic to virtualization as a tool.

Re: Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack

#72

Post by acdha »

hhw wrote: post #52
hp50g wrote: post #26

I think this only solves the cost problem. VMware is damn expensive for what it is. I'm not really a fan of virtualization myself. We canned it at our organisation a couple of years ago. We had 30 ESX/vSphere managed hosts across 3 data centers which hosted app servers, web front end servers, virtual load balancers and some other minor infrastructure. The final straw was the upgrade costs. Not only that we had proble…

Exactly. Virtualization/Cloud are only good for scaling down, not scaling up. The 20-30% hit to IO, additional latency of SAN vs local storage, management overhead, and added complexity to infrastructure more than negate any potential benefits of improved load distribution when you're dealing with applications that use anywhere near the capacity of physical servers.

> The 20-30% hit to IO

If you're seeing that, something is configured wrong. VMWare was 95+% of native disk and gigabit ethernet 5 years ago.

Re: Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack

#73

Post by acdha »

rektide wrote: post #67
hp50g wrote: post #61

Earlier quoted context omitted.

Its definitely 20-30% for realistic workloads using VM based tech on top I.e. CLR/JVM or a database engine which is realistic. This is on top of VMware. I can't speak for Xen. The outcome is pretty grim.

Ah, VMware is the problem, that is explains it. Parties that care muchly about fine performance margins apparently need to be using Xen or KVM or Illuminos then.

Can't speak for the parent poster's company but the numbers don't match my experience with VMware many years back. It's possible they've had a sharp regression but we were maxing out gigabit ethernet and local RAID arrays in 2006.

Re: Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack

#74

Post by anon987 »

babarock wrote: post #29

Slightly relevant: If you're interested in discovering OpenStack, you should check out devstack[1]. It's a bash script that (git) pulls different projects of openstack into a machine and install them. The script itself is very readable, thoroughly commented, on purpose. You're encouraged to read it while the whole thing installs (which can take several minutes). Feel free to launch it inside of a VM (preferably runni…

People can also check out http://trystack.org/ and SAIO (Swift all in One, a single node Swift cluster for development, testing, and learning) http://docs.openstack.org/developer/swift/development_saio.h...

Re: Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack

#75

Post by acdha »

daigoba66 wrote: post #18

What's wrong with VMware? Why would someone not choose VWware over something else?

Their pricing has gone up while the rest of the industry is trending down. This wouldn't be so bad except that most of their tools are very enterprise-y – clunky APIs, odd gaps in tooling, etc. Basically the hypervisor is developed by the A-team but the management tools and even components like the HA manager were obviously second-tier – big, convoluted snarls of daemons for something which should have been clean, simple and trustworthy. (i.e. I spent a few weeks on the phone attempting to convince second or third-level support that the HA system should also pay attention to the heartbeat checks rather than ethernet link status when deciding whether to use a network connection. The techs I talked to were under the impression that this was how it worked until I proved otherwise…)

I stopped supporting it in 2008 but at the time it was already apparent that they were trying to milk large companies rather than stay competitive – and that was before the price hikes.

Re: Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack

#76

Post by acdha »

samspenc wrote: post #22
daigoba66 wrote: post #18

What's wrong with VMware? Why would someone not choose VWware over something else?

In my experience, VMWare is great and has a nice GUI, which lead to early adoption. BUT: it has licensing fees for each copy. This can be a big bummer for companies like PayPal that have a fleet of servers - much easier to replace to replace all the VMWare installs and get rid of licensing costs to switch to an open-source VM solution. At the same time, I could be that OpenStack's API is superior to VMWare's. With a…

> At the same time, I could be that OpenStack's API is superior to VMWare's. With a fleet of servers, you want to automate as much as possible - meaning that at the end of the day, API trumps GUI.

It reflects the bizarre decision VMware made when they wanted only sell to the top end of the market: the biggest customers are also the most likely to have the resources and incentive to build tools around that API. The small to mid scale shops depend on the GUI tools far more.

Re: Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack

#77

Post by hp50g »

acdha wrote: post #72
hhw wrote: post #52

Earlier quoted context omitted.

Exactly. Virtualization/Cloud are only good for scaling down, not scaling up. The 20-30% hit to IO, additional latency of SAN vs local storage, management overhead, and added complexity to infrastructure more than negate any potential benefits of improved load distribution when you're dealing with applications that use anywhere near the capacity of physical servers.

> The 20-30% hit to IO If you're seeing that, something is configured wrong. VMWare was 95+% of native disk and gigabit ethernet 5 years ago.

Unless you are doing really heavy IO or CPU across multiple VMs on the same host which is likely if you have any load worth mentioning. There is a 20% to 30% difference if you run one process per VM or 8 processes on bare metal in an 8 core machine. We benchmarked it on known good configs optimised to bits.

Either the hypervisor scheduler is shit or the abstraction is costly. I reckon its down to the reduction in CPU cache availability and the IO mux.

This was HP kit end to end, VMware certified, debian stable on and off ESX 4.

Re: Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack

#78

Post by hp50g »

wmf wrote: post #68
hp50g wrote: post #64

Earlier quoted context omitted.

Processes are a form of virtualisation are they not? That's enough abstraction.

Originally, yes. Unfortunately Unix/Windows has accumulated a ton of global state (libfoo 3.7.1 not parallel installable with libfoo 3.6.4) and enterprise software is now so fragile that any alternative to VM sprawl is unthinkable.

We run our software (and other "enterprise" software) multiple instances per node and they are fully isolated. Most is in JVMs but we have C++ stuff with no problems.

Re: Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack

#79

Post by druiid »

hp50g wrote: post #62
incision wrote: post #59

Earlier quoted context omitted.

>It's snake oil. As opposed to a miracle elixir that cures all ills? It's a tool. Like most tools, it's best used when you thoroughly understand the challenge and how/why to apply that tool to it. The problems you've noted here seem more indicative of poor planning and understanding of the solution than any intrinsic deficiency of virtualization.

Its a shit tool that creates issues and costs fuckwads of cash. That is it. We understand it. The numbers sold are not the numbers gained. The entire infrastructure deployment was planned around virtualization, DC failover and resilience using VMware's guidelines and solutions (VMware were even paid to consult on this). It never delivered and doubled our administrative and licensing overhead.

VMWare sales from what I have heard, has a history of over-selling, over-promising and ultimately getting people to spend a crap-ton of money on their products.

Honestly I have to say, it sounds a bit like you guys got taken for a ride. I'd suggest sitting back, relaxing and taking a look at what other VM options there are out there, some of which don't have license fees attached to them.

When I first deployed virtualization there were really only two 'enterprise' ready solutions out there which were essentially VMWare or XenServer. Knowing about VMWare and their 'history' I chose XenServer. I haven't regretted the choice but they wouldn't be a fit for everyone.

These days there are multiple numbers of solutions out there to choose from, which Openstack may or may not be one for you... and that aren't going to whisper into your ear about how great their product is and how much money it will save you.

Re: Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack

#80

Post by samstave »

superuser2 wrote: post #65
3am wrote: post #25

I hate cloud headlines. What's the hypervisor? VMWare is a hypervisor (ESX) and management applications (vCenter/vSphere, vCloud Director, etc). Openstack is only the management applications. And VMWare contributes the most interesting part of OpenStack, anyway, which is the virtual networking based on Nicira's OpenFlow. My personal experience with OpenStack (admittedly late Diablo release timeframe) was that it was…

Rackspace Cloud Servers runs on OpenStack. It seems to work pretty well for them.

I would expect that it maybe heavily custom/staffed-up-to-be-configured-for-their-infra

I.E. - it likely didn't just work pretty well for them out of the box.

There is a difference between using a base technology which requires you have FTEs on staff to maintain it and grabbing an off-the-shelf app to help run your systems.

Post reply on HN