Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The AWS EC2 Windows Secret Sauce

technodrone.blogspot.com

Post reply on HN
Print view View original

21–30 of 44 posts

Re: The AWS EC2 Windows Secret Sauce

#21

Post by pjc50 »

Twirrim wrote: post #16

> Seriously though - Windows images are big - absolutely massive compared to a Linux image - we are talking 30 times larger (on the best of days) so copying these large images to the hypervisor nodes takes time. No... just no. Disks aren't local to hypervisors. There's no copying going to be taking place. EC2 instances are provisioned using EBS volumes, which aren't going to be local to the instance itself. EBS is li…

> Windows goes through a mandatory process whereby it identifies hardware, installs and configures drivers etc. etc. etc. You can inject a script to carry out the process of things like setting up a user, and generating a password, but that's fairly minor on the scale of things. This first process requires a reboot. There's no escaping it.

This is the "OOBE" (out of box experience) phase. I'd have thought they'd just skip it and provision you a pre-warmed image .. but I guess because there's a license/activation dependency they can't do that?

I wonder how many MWh could be saved by Microsoft adding a "acquire cloud volume license at boot" mode. Or shoving the licensing/uniquification requirements into the platform TPM.

Re: The AWS EC2 Windows Secret Sauce

#22

Post by maxaf »

gtsteve wrote: post #15

This is an interesting thought; I use Windows instances myself but I use a custom AMI built using Packer on our CI server. Presumably Amazon doesn't have a pool of my custom AMI images lying around. So I wonder if the custom AMI is actually stored as a layer on top of the source Windows AMI and applied to the instance from the pool before it is made available to me. Alternatively, it means I'm missing out on an optim…

A couple of jobs ago I have spent plenty of time optimizing a Packer-based AMI pipeline that was similar in spirit to that which you've described. It was all quite frustrating, but then the job got yanked from underneath me, so I never finished solving the problem.

The article contains a screenshot[1] of a response from AWS support indicating that only EC2's own AMIs benefit from this pooling optimization. It follows that custom AMIs must take the startup time hit all the way.

[1]: https://maishsk.com/blog/images/20190307_aws_ec2_secret/foru...

Re: The AWS EC2 Windows Secret Sauce

#23

Post by gtsteve »

maxaf wrote: post #22
gtsteve wrote: post #15

This is an interesting thought; I use Windows instances myself but I use a custom AMI built using Packer on our CI server. Presumably Amazon doesn't have a pool of my custom AMI images lying around. So I wonder if the custom AMI is actually stored as a layer on top of the source Windows AMI and applied to the instance from the pool before it is made available to me. Alternatively, it means I'm missing out on an optim…

A couple of jobs ago I have spent plenty of time optimizing a Packer-based AMI pipeline that was similar in spirit to that which you've described. It was all quite frustrating, but then the job got yanked from underneath me, so I never finished solving the problem. The article contains a screenshot[1] of a response from AWS support indicating that only EC2's own AMIs benefit from this pooling optimization. It follows…

Ah, I had missed that! Thank you, that's very helpful to know. I always learn something new every day here.

Re: The AWS EC2 Windows Secret Sauce

#24

Post by sl1ck731 »

Zombiethrowaway wrote: post #14

I used Windows instances a few years ago. Beyond the slow start, once started, frequently the CPU would stay stuck at very low %, and my tasks would run very slowly. Eventually I would get to 100%, but it could often take 10 minutes. What I learned from those pains is how to use Linux in the Cloud.

I've experienced this lately with a variety of Amazon Windows images. For example I will boot a 2016 image from this year vs one from last year and last year's will be significantly faster on the same hardware.

Re: The AWS EC2 Windows Secret Sauce

#25

Post by mc32 »

briffle wrote: post #20
scarface74 wrote: post #5

I’ve been a Microsoft developer for over two decades. But once I started architecting solutions on AWS and seeing the Windows Tax first hand - in terms of resource requirements and licensing costs - I started trying to avoid Windows like the plague. Also the true costs of infrastructure became my problem - accounting can see exactly what a solution costs - instead of some amorphous cost in the IT budget.

I really like the better visibility into the true costs. Too many times in the past, a team had a budget for the software they needed to deploy a project, but the Virtualization software, storage array, etc, were all out of the IT budget. that makes the IT budget look bloated, and hard to explain, and ripe for being seen as a 'cost center'.

That’s too bad and unfortunate of some companies who see IT as a black hole. Imagine having an R&D dept which could incur lots of costs and little result individually but as a group produce good licensable IP. Sometimes you have to realize some things are a cost of doing business.

Re: The AWS EC2 Windows Secret Sauce

#26

Post by scarface74 »

mc32 wrote: post #25
briffle wrote: post #20

Earlier quoted context omitted.

I really like the better visibility into the true costs. Too many times in the past, a team had a budget for the software they needed to deploy a project, but the Virtualization software, storage array, etc, were all out of the IT budget. that makes the IT budget look bloated, and hard to explain, and ripe for being seen as a 'cost center'.

That’s too bad and unfortunate of some companies who see IT as a black hole. Imagine having an R&D dept which could incur lots of costs and little result individually but as a group produce good licensable IP. Sometimes you have to realize some things are a cost of doing business.

It's not that they don't realize it. The R&D department doesn't have to be accountable for costs. The R&D department isn't incentivized to control costs because it isn't our problem. Proper tagging of resources shines a light on the R&D department and the business knows exactly what our projects' infrastructure cost is.

Re: The AWS EC2 Windows Secret Sauce

#27

Post by tachyonbeam »

sl1ck731 wrote: post #24
Zombiethrowaway wrote: post #14

I used Windows instances a few years ago. Beyond the slow start, once started, frequently the CPU would stay stuck at very low %, and my tasks would run very slowly. Eventually I would get to 100%, but it could often take 10 minutes. What I learned from those pains is how to use Linux in the Cloud.

I've experienced this lately with a variety of Amazon Windows images. For example I will boot a 2016 image from this year vs one from last year and last year's will be significantly faster on the same hardware.

Any chance that might have to do with patches for the spectre vulnerability taking a performance toll?

https://en.wikipedia.org/wiki/Spectre_(security_vulnerabilit...

Re: The AWS EC2 Windows Secret Sauce

#28

Post by jedieaston »

pjc50 wrote: post #21
Twirrim wrote: post #16

> Seriously though - Windows images are big - absolutely massive compared to a Linux image - we are talking 30 times larger (on the best of days) so copying these large images to the hypervisor nodes takes time. No... just no. Disks aren't local to hypervisors. There's no copying going to be taking place. EC2 instances are provisioned using EBS volumes, which aren't going to be local to the instance itself. EBS is li…

> Windows goes through a mandatory process whereby it identifies hardware, installs and configures drivers etc. etc. etc. You can inject a script to carry out the process of things like setting up a user, and generating a password, but that's fairly minor on the scale of things. This first process requires a reboot. There's no escaping it. This is the "OOBE" (out of box experience) phase. I'd have thought they'd just…

OOBE occurs after this, the hardware identification phase is what takes the longest time. It has to reboot to load drivers and take 5-10 minutes to identify what hardware in the machine needs to have drivers, etc. Once you get to the blue "Just a moment..." screen, you can inject a script to activate Windows and get to the desktop.

Re: The AWS EC2 Windows Secret Sauce

#29

Post by vegardx »

Zombiethrowaway wrote: post #14

I used Windows instances a few years ago. Beyond the slow start, once started, frequently the CPU would stay stuck at very low %, and my tasks would run very slowly. Eventually I would get to 100%, but it could often take 10 minutes. What I learned from those pains is how to use Linux in the Cloud.

Seems like you were using T2 instances which have a low baseline performance and burst credits. I would imagine that you quickly run out of credits on some of the smaller instance types after creation, given how lengthy and costly (in terms of CPU usage) the instance creation and boot process is.

Re: The AWS EC2 Windows Secret Sauce

#30

Post by sudovancity »

>Now that I have got your attention with a catchy title - let me share with some of my thoughts regarding how AWS shines and how much your experience as a customer matters.

This is the one guaranteed way to turn me off to whatever you're going to talk about in the article.

Post reply on HN