Making Facebook’s software infrastructure more energy efficient with Autoscale • Hacker News Board

Re: Making Facebook’s software infrastructure more energy efficient with Autoscale

#21

Post by reeze_xia » Sat, Aug 09, 2014, 4:17 PM UTC

otterley wrote: post #20

reeze_xia wrote: post #14

Earlier quoted context omitted.

I am curious why Facebook don't use? If they do use virtualization, the fact, 0 request lead to low power does not correct any more, for example linux container, other container may accept requests, if so they have to dispatch request cross all containers and servers.

The Achilles heel of virtualization is networking. All of the hypervisors out there (VMWare, Xen, KVM) have user-space software switch implementations that dramatically reduce the throughput of TCP session creation. As a consequence you lose a significant amount of hardware potential to serve HTTP connections.

How about lightweight virtualization like linux container, docker?

Re: Making Facebook’s software infrastructure more energy efficient with Autoscale

#22

Post by otterley » Sat, Aug 09, 2014, 6:17 PM UTC

reeze_xia wrote: post #21

otterley wrote: post #20

Earlier quoted context omitted.

The Achilles heel of virtualization is networking. All of the hypervisors out there (VMWare, Xen, KVM) have user-space software switch implementations that dramatically reduce the throughput of TCP session creation. As a consequence you lose a significant amount of hardware potential to serve HTTP connections.

How about lightweight virtualization like linux container, docker?

That's not virtualization; it's namespace isolation. There's a small performance impact if you're using NAT, but otherwise the kernel networking stack is used, so there's no performance penalty.

Re: Making Facebook’s software infrastructure more energy efficient with Autoscale

#23

Post by vicaya » Sun, Aug 10, 2014, 12:40 AM UTC

SEJeff wrote: post #3

praseodym wrote: post #2

This is one of these things where virtualisation can help even more. For example, VMware can dynamically put servers in standby mode when demand is low and power them up again when needed: http://www.vmware.com/products/vsphere/features/drs-dpm

You know that would be trivial to do with bare metal and out of band management cards like Dell Dracs, IBM RSA cards, HP ILOs, or generic IPMI BMCs, right? Virtualization doesn't really add much of anything for that specific problem other than increased context switching and slightly lower performance. Disclaimer: building this type of thing (on bare metal) is a chunk of my day job. I see it as unbelievably trivial.…

"Virtualization doesn't really add much of anything for that specific problem other than increased context switching and slightly lower performance."

This is BS. What if you have 3 physical servers with 30% utilization? DRS can _seamlessly_ consolidate _arbitrary_ application VMs to one server and shutdown the rest. With bare metal, only certain specifically designed workloads (stateless web farms and some distributed systems etc.) can be moved easily.

It seems that dismissing virtualization out of sheer ignorance is a fad these days. Virtualization provides important hardware abstraction to a much wider variety of workloads.

Re: Making Facebook’s software infrastructure more energy efficient with Autoscale

#24

Post by vicaya » Sun, Aug 10, 2014, 12:43 AM UTC

otterley wrote: post #20

reeze_xia wrote: post #14

Earlier quoted context omitted.

I am curious why Facebook don't use? If they do use virtualization, the fact, 0 request lead to low power does not correct any more, for example linux container, other container may accept requests, if so they have to dispatch request cross all containers and servers.

The Achilles heel of virtualization is networking. All of the hypervisors out there (VMWare, Xen, KVM) have user-space software switch implementations that dramatically reduce the throughput of TCP session creation. As a consequence you lose a significant amount of hardware potential to serve HTTP connections.

This is wrong as well. VMware standard and distributed virtual switch is in hypervisor kernel.

Re: Making Facebook’s software infrastructure more energy efficient with Autoscale

#25

Post by wmf » Sun, Aug 10, 2014, 1:12 AM UTC

otterley wrote: post #20

reeze_xia wrote: post #14

Earlier quoted context omitted.

I am curious why Facebook don't use? If they do use virtualization, the fact, 0 request lead to low power does not correct any more, for example linux container, other container may accept requests, if so they have to dispatch request cross all containers and servers.

The Achilles heel of virtualization is networking. All of the hypervisors out there (VMWare, Xen, KVM) have user-space software switch implementations that dramatically reduce the throughput of TCP session creation. As a consequence you lose a significant amount of hardware potential to serve HTTP connections.

We just did some benchmarks on this topic: https://news.ycombinator.com/item?id=8146536

Re: Making Facebook’s software infrastructure more energy efficient with Autoscale

#26

Post by wmf » Sun, Aug 10, 2014, 1:17 AM UTC

mschuster91 wrote: post #16

bhouston wrote: post #15

Vfx companies have software that turn off renddr farm machines in periods of low load. I helped write software that did this backin 2006. Basically, we found it was possible to shut down machines in periods of low load and then use "Wake On Lan" to start them up once load picked up again. I am unsure if the on-off power cycling reduces machine longevity. Might be something worth exploring at Facebook.

> I am unsure if the on-off power cycling reduces machine longevity. Possibly the disks are the most vulnerable components. I wonder if it would be possible in software to shut down the CPUs and have only disks + network running...

In a diurnal cycle like Facebook's, you'd have one start-stop per day, which should be well within the rated specs of hard disks. A few years back I looked at the idea of treating disk lifetime as a resource and explicitly managing it: http://dx.doi.org/10.1109/MSST.2011.5937221

That's assuming that the servers have disks at all, which they probably shouldn't.

Re: Making Facebook’s software infrastructure more energy efficient with Autoscale

#27

Post by otterley » Sun, Aug 10, 2014, 3:28 PM UTC

vicaya wrote: post #24

otterley wrote: post #20

Earlier quoted context omitted.

The Achilles heel of virtualization is networking. All of the hypervisors out there (VMWare, Xen, KVM) have user-space software switch implementations that dramatically reduce the throughput of TCP session creation. As a consequence you lose a significant amount of hardware potential to serve HTTP connections.

This is wrong as well. VMware standard and distributed virtual switch is in hypervisor kernel.

How's VMWare's TCP new-session-switching-rate compared to Xen's OVS? Any benchmarks showing a significant improvement?

Re: Making Facebook’s software infrastructure more energy efficient with Autoscale

#28

Post by reeze_xia » Mon, Aug 11, 2014, 2:46 AM UTC

otterley wrote: post #22

reeze_xia wrote: post #21

Earlier quoted context omitted.

How about lightweight virtualization like linux container, docker?

That's not virtualization; it's namespace isolation. There's a small performance impact if you're using NAT, but otherwise the kernel networking stack is used, so there's no performance penalty.

Yes, indeed. I mean the optimization of energy saving seems not suitable for the condition of namespace isolation, since you can't control the other containers requests. if we have to, we need to dispatch request from server perspective but not container.