The sorry state of server utilization and the post-hypervisor era (2013) • Hacker News Board

Re: The sorry state of server utilization and the post-hypervisor era (2013)

#21

Post by parasubvert » Sun, May 17, 2015, 3:38 PM UTC

dsr_ wrote: post #13

Which of these hypothetical situations is more realistic? CEO: "I see that we had 99.994% uptime for the last six months, and we came in very close to the forecasted budget. Well done, engineers!" CEO: "I see that we had 99.9% efficient usage for the last six months, and we reduced our budget. Well done, engineers!" Neither scenario is realistic, of course. Uptime is nice and efficiency is nice and budgets are nice,…

I've worked with the executives of some large banks, telecoms, and transportation companies. The CEO and board generally only has held the IT team accountable to budgetary performance and risk (uptime, intrusion, regulatory) metrics. The only IT impact on customer sat is uptime, by the traditional view.

One bank IT group I know that reports on uptime to their business partners relative to operating expense, prints the charts and graphs on plotter paper weekly and posts them in the cafeteria. Most of their bonus is directly tied to those numbers. So, "cut costs and keep me up".

Delivery IT groups are very rarely measured by customer satisfaction, they're measured by project and budget performance to baseline (on time, on budget, etc). Customer sat is the responsibility of the business partners that drive the requirements, programs, etc.

This this effective? Not really. If they recognized Lean product development principles they'd incentivize everything by end-to-end cost of delay first, and risk reduction second.

Re: The sorry state of server utilization and the post-hypervisor era (2013)

#22

Post by lsc » Sun, May 17, 2015, 3:45 PM UTC

otterley wrote: post #20

lsc wrote: post #15

"A post-hypervisor world " lol. I've been predicting a backlash against this virtualization hype since 2005, and this is the first time I've heard anyone else mention anything like it. Of course, if you had told me in 2005 that we would be switching from hypervisors back to containers, I would have broke down crying. Is our industry run by masochists? or just the inexperienced, who don't know any better?

To be fair, containers aren't a virtualization solution; they're more of a packaging mechanism.

that is... a healthy way to look at it.

My experience has been that using containers to go multi-tenant leads only to misery and pain.

But it does seem a reasonable-ish way to handle packaging, though I have less experience with that use case. It does seem like it would work, assuming you still have a way to update everything, and assuming everything is happy with the same kernel.

Re: The sorry state of server utilization and the post-hypervisor era (2013)

#23

Post by lsc » Sun, May 17, 2015, 3:56 PM UTC

parasubvert wrote: post #18

lsc wrote: post #15

"A post-hypervisor world " lol. I've been predicting a backlash against this virtualization hype since 2005, and this is the first time I've heard anyone else mention anything like it. Of course, if you had told me in 2005 that we would be switching from hypervisors back to containers, I would have broke down crying. Is our industry run by masochists? or just the inexperienced, who don't know any better?

This was written by a VC hoping that Docker is going to be worth more than VMware. I suspect he may be disappointed.

A lot of the value of vmware is in the sales channels. Why use VMware rather than QEMU/KVM? it used to be that VMware came with support. But now that KVM is owned by RedHat, which in my experience, gives way better than average support? yeah.

but, yeah. Docker doesn't solve the "take this ancient rack of failing servers and consolidate them down to one server... without updating the software" use case that VMware is so often used for.

Re: The sorry state of server utilization and the post-hypervisor era (2013)

#24

Post by parasubvert » Sun, May 17, 2015, 3:59 PM UTC

LamaOfRuin wrote: post #16

parasubvert wrote: post #9

Earlier quoted context omitted.

at Google's scale, one doesn't need a lot of incentive to improve utilization. Every IT shop has wanted the cost reduction of improved utilization since the dawn of the PC era. The difference is in process. Google's approach to workload placement is automated by software, driven by engineering decisions and data. Many IT shops' placement is political (new servers = new capital = power).

At Google's scale you need more much incentive to get anything done. This is even more true when it is something that will touch every division, product, and service. What every IT shop wants doesn't necessarily relate in any straightforward way to what any IT shop invests resources in getting. Every IT shop prioritizes many other things above utilization (and are right to do so). All decisions, engineering or otherw…

All decisions are political (ie. Power interests), but not all orginzations are configured to be primarily driven by power. This is especially true for young organizations, or those that have gone through a cycle of renewal.

Google decided early on to drive towards an operational architecture that allows individuals to act at scale on their infrastructure. A developer deploys into production, it launches thousands of new containers and disposes thousands of old containers. A batch job is run, same thing. Deploying services is uniform across the board. Thus, optimizing utilization through improved container scheduling is something that the core site reliability engineering team could do independently of individual services.

Google's early adoption of data center sized computing by Hozle & team was unique, along with Amazon's CEO-diktat move to decentralized service-oriented architecture, or Netflix's rewrite and move to cloud. Which is why you have articles like this, written by a VC, that want to repackage this thinking and sell it back to old school IT.

Re: The sorry state of server utilization and the post-hypervisor era (2013)

#25

Post by parasubvert » Sun, May 17, 2015, 4:16 PM UTC

lsc wrote: post #23

parasubvert wrote: post #18

Earlier quoted context omitted.

This was written by a VC hoping that Docker is going to be worth more than VMware. I suspect he may be disappointed.

A lot of the value of vmware is in the sales channels. Why use VMware rather than QEMU/KVM? it used to be that VMware came with support. But now that KVM is owned by RedHat, which in my experience, gives way better than average support? yeah. but, yeah. Docker doesn't solve the "take this ancient rack of failing servers and consolidate them down to one server... without updating the software" use case that VMware is…

Why use VMware? Is indeed the existential question that's facing them. For now, it's because many IT shops can't wrap their heads around the alternatives or justify the switch. Lack of skills (lots of Windoes centric shops), deep love for DRS/Vmotion/HA, deep support for fibre channel setups, etc. Btw this is arguably why VMware announced Photon recently, to go after RedHat. Eat at their Linux monopoly.

That said, VMware did basically invent x86 virtualization as we know it today, and that's justified the many billions in wealth it has generated to date. Docker is (so far) a registry and a CLI wrapper around a Linux kernel feature. It can and will be more, but it's not clear what.

Re: The sorry state of server utilization and the post-hypervisor era (2013)

#26

Post by LamaOfRuin » Sun, May 17, 2015, 6:03 PM UTC

parasubvert wrote: post #24

LamaOfRuin wrote: post #16

Earlier quoted context omitted.

At Google's scale you need more much incentive to get anything done. This is even more true when it is something that will touch every division, product, and service. What every IT shop wants doesn't necessarily relate in any straightforward way to what any IT shop invests resources in getting. Every IT shop prioritizes many other things above utilization (and are right to do so). All decisions, engineering or otherw…

All decisions are political (ie. Power interests), but not all orginzations are configured to be primarily driven by power. This is especially true for young organizations, or those that have gone through a cycle of renewal. Google decided early on to drive towards an operational architecture that allows individuals to act at scale on their infrastructure. A developer deploys into production, it launches thousands of…

> Thus, optimizing utilization through improved container scheduling is something that the core site reliability engineering team could do independently of individual services.

But is that something it is known they prioritized, or was there perhaps more interest in optimizing the efficiency of deploying thousands of containers on every deploy, across data centers, with reliable testing, without killing in flight processing, and scaling for subsecond response to bursty demand? Who sets the priorities for what is most important, and how much of one they're willing to sacrifice to improve physical utilization?

I have absolutely no doubt they had as many resources as any other company dedicated to finely tuning their data centers and related infrastructure. I question whether they had the same motivation as a company like Amazon (who was deriving direct profit from selling this resource) to prioritize the optimization of utilization.