Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack

forbes.com

Post reply on HN
Print view View original

81–90 of 128 posts

Re: Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack

#81

Post by druiid »

acdha wrote: post #73
rektide wrote: post #67

Earlier quoted context omitted.

Ah, VMware is the problem, that is explains it. Parties that care muchly about fine performance margins apparently need to be using Xen or KVM or Illuminos then.

Can't speak for the parent poster's company but the numbers don't match my experience with VMware many years back. It's possible they've had a sharp regression but we were maxing out gigabit ethernet and local RAID arrays in 2006.

Well, don't confuse I/O with throughput here. You can look at performance numbers for just about anything and tweak one direction or the other.

For instance it's easy to make a benchmark showing huge throughput to any given storage solution (and many NAS providers sell on this basis), but your I/O might be terrible because to get that throughput you're maxing the CPU (etc.). Likewise, you can change your benchmark and show high I/O, but the throughput is 'terrible'.

Re: Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack

#82

Post by aliguori »

3am wrote: post #25

I hate cloud headlines. What's the hypervisor? VMWare is a hypervisor (ESX) and management applications (vCenter/vSphere, vCloud Director, etc). Openstack is only the management applications. And VMWare contributes the most interesting part of OpenStack, anyway, which is the virtual networking based on Nicira's OpenFlow. My personal experience with OpenStack (admittedly late Diablo release timeframe) was that it was…

http://www.slideshare.net/openstackindia/openstack-at-paypal...

Has more details. Looks like it's KVM.

Re: Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack

#83

Post by druiid »

hp50g wrote: post #77
acdha wrote: post #72

Earlier quoted context omitted.

> The 20-30% hit to IO If you're seeing that, something is configured wrong. VMWare was 95+% of native disk and gigabit ethernet 5 years ago.

Unless you are doing really heavy IO or CPU across multiple VMs on the same host which is likely if you have any load worth mentioning. There is a 20% to 30% difference if you run one process per VM or 8 processes on bare metal in an 8 core machine. We benchmarked it on known good configs optimised to bits. Either the hypervisor scheduler is shit or the abstraction is costly. I reckon its down to the reduction in CPU…

Honest question here. First: What year exactly was this deployment put in place? At least for Intel their CPU performance for VM has increased SUBSTANTIALLY even in the last couple years. I remember taking some VM hosts from some of their first (or second? Can't remember for sure) gen processors to the 5500 series. It was like night and day. It is even more so for the e5 series.

Second: Why did you implement a large-scale VMWare install without either having a testing period before sinking contract dollars and license costs into it, or at least having contract terms to opt-out if their claims didn't match reality?

Re: Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack

#84

Post by hhw »

hp50g wrote: post #61
druiid wrote: post #54

Earlier quoted context omitted.

20-30% hit in IO? It hasn't been at that level for a long time. With new KVM versions and good Intel processors there is a performance hit of as low as 5% these days. That's not to necessarily say your particular workload will see that, but 10% is generally an 'at-worst' level at this point.

Its definitely 20-30% for realistic workloads using VM based tech on top I.e. CLR/JVM or a database engine which is realistic. This is on top of VMware. I can't speak for Xen. The outcome is pretty grim.

Exactly. I often see claims of 5-10%, but I've yet to see any reliable set of benchmarks done with those results. Too often, people are using dd and testing throughput instead of actual IOPS. Even the benchmarks that show 20%+ tend to be skewed in favour of virtualization, as they tend to be run with a single VM instead of multiple VM's.

Even if there was 0% performance penalty from virtualization, you'd still see suboptimal allocation of hardware resources just from trying to take an abstracted view of the hardware. Different applications have different performance profiles. You either end up with overbuilt hardware to support the virtualization environment and the different performance profiles of the different applications, or with multiple VM's for the same application on the same hardware which is totally unnecessary overhead. Virtualization is just not meant for large scale.

Re: Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack

#85

Post by spikels »

rektide wrote: post #70
sc68cal wrote: post #38

Earlier quoted context omitted.

You can also use Vagrant+Devstack[1] to get started even quicker. https://github.com/bcwaldon/vagrant_devstack

So, what, now you can virtualize while your virtualize? I thought the point of OpenStack was to be a way to manage VMs, isn't that also much of what Vagrant does? I'd tend to avoid mixing the streams & go for a single node management system rather than attempt two at once- were it me. But I also am not invested in Vagrant, which people are free to enjoy, so if they can happily run both VM managers at once, so be it.…

Testing new software in a VM is done primarily because (as stated above) you "avoid polluting your current system". When you are done or if anything goes wrong you can just trash the VM. This is standard practice in a variety of situations and has other benefits.

The only twist here is the software you are testing happens to be virtualization management software. Make sense?

Re: Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack

#86

Post by zwily »

rektide wrote: post #70
sc68cal wrote: post #38

Earlier quoted context omitted.

You can also use Vagrant+Devstack[1] to get started even quicker. https://github.com/bcwaldon/vagrant_devstack

So, what, now you can virtualize while your virtualize? I thought the point of OpenStack was to be a way to manage VMs, isn't that also much of what Vagrant does? I'd tend to avoid mixing the streams & go for a single node management system rather than attempt two at once- were it me. But I also am not invested in Vagrant, which people are free to enjoy, so if they can happily run both VM managers at once, so be it.…

One example of where this is very useful is for people not running Linux on their workstation. If I want to play with OpenStack or Docker or whatever, a VM is a nice playground for it, and Vagrant makes convenient VMs.

Re: Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack

#87

Post by 3am »

viraptor wrote: post #47
3am wrote: post #36

Earlier quoted context omitted.

I've kept up a little on OpenStack since then and know people that work with the technology. I'm sure the keystone integrated auth system (or whatever it's called now) really improved things, and it seems like people are making really progress with pushbutton deployments using the usual config/orchestration suspects. Congrats! That said I know more people would would be content to manage ESX with ovftool or some home…

What do you mean by "linux kernel without the usual userspace tools"? Openstack runs on a complete system with all the tools you want to install, not some bare kernel machine.

"OpenStack is like you think about the linux kernel. Most people want a distro, not raw source to setup their environments"

Since he was comparing the relationship between a hypervisor and Openstack to that between the Linux kernel and the userspace tools, and I was just commenting on that comparison. That is, which I know people that use hypervisors without Openstack (or other virtualization management applications like Cloudstack or vCenter), I don't know anyone that uses the kernel on its own. Sorry, I thought that was clear but I guess it was not.

Re: Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack

#88

Post by IheartApplesDix »

hp50g wrote: post #64
IheartApplesDix wrote: post #50

Earlier quoted context omitted.

Virtualization is a requirement for any kind of serious dynamic infrastructure.

Processes are a form of virtualisation are they not? That's enough abstraction.

Virtualization is a form of process so yes.

Re: Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack

#89

Post by hp50g »

druiid wrote: post #83
hp50g wrote: post #77

Earlier quoted context omitted.

Unless you are doing really heavy IO or CPU across multiple VMs on the same host which is likely if you have any load worth mentioning. There is a 20% to 30% difference if you run one process per VM or 8 processes on bare metal in an 8 core machine. We benchmarked it on known good configs optimised to bits. Either the hypervisor scheduler is shit or the abstraction is costly. I reckon its down to the reduction in CPU…

Honest question here. First: What year exactly was this deployment put in place? At least for Intel their CPU performance for VM has increased SUBSTANTIALLY even in the last couple years. I remember taking some VM hosts from some of their first (or second? Can't remember for sure) gen processors to the 5500 series. It was like night and day. It is even more so for the e5 series. Second: Why did you implement a large-…

2008. TBH not sure what CPUs we had in there. Kit has been scrapped now. I wouldn't bother going through it again. We now have a standard half and full rack which we can purchase and deploy quickly for an installation so there are no plans to piss any more time and cost up the wall.

We did have a testing period which was unfortunately run by a Muppet who decided the loss was acceptable. Muppet now no longer works for organisation.

Re: Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack

#90

Post by sgloutnikov »

I think VMWare is going to be in some trouble in the next few years. A lot of opensource projects are picking up speed. One of my favorites, oVirt (http://www.ovirt.org) especially. Red Hat mainly runs the project, and is pretty much the open source version of RHEV.
Post reply on HN