Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making Facebook’s software infrastructure more energy efficient with Autoscale

code.facebook.com

Post reply on HN
Print view View original

11–20 of 28 posts

Re: Making Facebook’s software infrastructure more energy efficient with Autoscale

#11

Post by wmf »

thrownaway2424 wrote: post #10
wmf wrote: post #9

Earlier quoted context omitted.

Real servers can't be bought without IPMI and AFAIK the BMC cannot be turned off, so it's probably not worth worrying about BMC power if there's nothing you can do about it.

Sure, but as you're aware facebook, google, et al don't buy "real servers", they buy servers that actually meet their requirements. That's why "real vendors" like HP have missed the boat on selling millions of servers into the cloud.

Speaking of Facebook specifically, the evolution is interesting. They replaced BMCs with the reboot-on-LAN hack but then their next motherboard version had BMCs again. It would be interesting to hear the story behind that.

Re: Making Facebook’s software infrastructure more energy efficient with Autoscale

#12

Post by SEJeff »

thrownaway2424 wrote: post #7
devicenull wrote: post #5

Earlier quoted context omitted.

Getting IPMI controllers to do what you want is anything but trivial :)

Seriously. A person who would advocate using IPMI at scale has either never owned an IPMI card or has never worked at scale or both. The just don't work, and they erase whatever power savings you're trying to achieve.

I actually advocated using the vendor specific ones with the BMC being if you use shitty hardware. It works ok, but is absolutely suboptimal.

Re: Making Facebook’s software infrastructure more energy efficient with Autoscale

#13

Post by devicenull »

thrownaway2424 wrote: post #7
devicenull wrote: post #5

Earlier quoted context omitted.

Getting IPMI controllers to do what you want is anything but trivial :)

Seriously. A person who would advocate using IPMI at scale has either never owned an IPMI card or has never worked at scale or both. The just don't work, and they erase whatever power savings you're trying to achieve.

Although... if you have the engineering resources of Facebook, you can write your own IPMI software and probably get it working pretty well. They're all just embedded ARM systems after all..

I ran out of motivation, but I did manage to fix up some of the SuperMicro IPMI firmware: https://github.com/devicenull/supermicro_ipmi_firmware

Re: Making Facebook’s software infrastructure more energy efficient with Autoscale

#14

Post by reeze_xia »

wmf wrote: post #4
SEJeff wrote: post #3

Earlier quoted context omitted.

You know that would be trivial to do with bare metal and out of band management cards like Dell Dracs, IBM RSA cards, HP ILOs, or generic IPMI BMCs, right? Virtualization doesn't really add much of anything for that specific problem other than increased context switching and slightly lower performance. Disclaimer: building this type of thing (on bare metal) is a chunk of my day job. I see it as unbelievably trivial.…

If you want to power manage your pets, VMware makes sense. But yeah, there's a reason Facebook doesn't use virtualization.

I am curious why Facebook don't use?

If they do use virtualization, the fact, 0 request lead to low power does not correct any more, for example linux container, other container may accept requests, if so they have to dispatch request cross all containers and servers.

Re: Making Facebook’s software infrastructure more energy efficient with Autoscale

#15

Post by bhouston »

Vfx companies have software that turn off renddr farm machines in periods of low load. I helped write software that did this backin 2006.

Basically, we found it was possible to shut down machines in periods of low load and then use "Wake On Lan" to start them up once load picked up again.

I am unsure if the on-off power cycling reduces machine longevity.

Might be something worth exploring at Facebook.

Re: Making Facebook’s software infrastructure more energy efficient with Autoscale

#16

Post by mschuster91 »

bhouston wrote: post #15

Vfx companies have software that turn off renddr farm machines in periods of low load. I helped write software that did this backin 2006. Basically, we found it was possible to shut down machines in periods of low load and then use "Wake On Lan" to start them up once load picked up again. I am unsure if the on-off power cycling reduces machine longevity. Might be something worth exploring at Facebook.

> I am unsure if the on-off power cycling reduces machine longevity.

Possibly the disks are the most vulnerable components. I wonder if it would be possible in software to shut down the CPUs and have only disks + network running...

Re: Making Facebook’s software infrastructure more energy efficient with Autoscale

#17

Post by meatmanek »

You could probably get a similar effect by using HAProxy's "balance first" algorithm, which chooses the first available server with an available connection slot (as defined by maxconn). If you did this, you'd want to set maxconn pretty conservatively.

Re: Making Facebook’s software infrastructure more energy efficient with Autoscale

#18

Post by donavanm »

mschuster91 wrote: post #16
bhouston wrote: post #15

Vfx companies have software that turn off renddr farm machines in periods of low load. I helped write software that did this backin 2006. Basically, we found it was possible to shut down machines in periods of low load and then use "Wake On Lan" to start them up once load picked up again. I am unsure if the on-off power cycling reduces machine longevity. Might be something worth exploring at Facebook.

> I am unsure if the on-off power cycling reduces machine longevity. Possibly the disks are the most vulnerable components. I wonder if it would be possible in software to shut down the CPUs and have only disks + network running...

A bit late to the party on this one. Check out cpu "c states" and things like intels "speedstep." Modern cpus can reduce/shutdown power to individual packages and cores. This can reduce power consumption from a hundred tdp to tens of tdp.

The downside is latency associated with changing state. Depending on the change it can be hundreds or thousands of micros to go through these states. On a server workload this can introduce huge latency outliers as a request blocks on a core to wake up.

Re: Making Facebook’s software infrastructure more energy efficient with Autoscale

#19

Post by mschuster91 »

donavanm wrote: post #18
mschuster91 wrote: post #16

Earlier quoted context omitted.

> I am unsure if the on-off power cycling reduces machine longevity. Possibly the disks are the most vulnerable components. I wonder if it would be possible in software to shut down the CPUs and have only disks + network running...

A bit late to the party on this one. Check out cpu "c states" and things like intels "speedstep." Modern cpus can reduce/shutdown power to individual packages and cores. This can reduce power consumption from a hundred tdp to tens of tdp. The downside is latency associated with changing state. Depending on the change it can be hundreds or thousands of micros to go through these states. On a server workload this can i…

I was more after "shut down everything except the disks" to have the absolute minimum running and the disks spinning to reduce wear on the engines.

If you're shutting down, you have massive latency anyway but if it's possible to at least save all the power not required for keeping disks up that 'd be great.

Re: Making Facebook’s software infrastructure more energy efficient with Autoscale

#20

Post by otterley »

reeze_xia wrote: post #14
wmf wrote: post #4

Earlier quoted context omitted.

If you want to power manage your pets, VMware makes sense. But yeah, there's a reason Facebook doesn't use virtualization.

I am curious why Facebook don't use? If they do use virtualization, the fact, 0 request lead to low power does not correct any more, for example linux container, other container may accept requests, if so they have to dispatch request cross all containers and servers.

The Achilles heel of virtualization is networking. All of the hypervisors out there (VMWare, Xen, KVM) have user-space software switch implementations that dramatically reduce the throughput of TCP session creation. As a consequence you lose a significant amount of hardware potential to serve HTTP connections.
Post reply on HN