Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Azure has run out of compute – anyone else affected?

news.ycombinator.com

Post reply on HN
Print view View original

291–300 of 359 posts

Re: Ask HN: Azure has run out of compute – anyone else affected?

#291

Post by Twirrim »

femto113 wrote: post #278
Twirrim wrote: post #261

Earlier quoted context omitted.

So does google, so does azure etc. etc. https://cloud.google.com/spot-vms , https://azure.microsoft.com/en-us/products/virtual-machines/... Spot instances exist just to try to turn over-provisions in to not a complete loss. You're at least making some money from your mistake. edit: You should consider "spot instances" in general to be a failure as far as a cloud provider is concerned. It means you've got your guesses…

Cloud providers aren't guessing at demand to plan capacity, they're literally building new data centers and then wheeling new racks into them as fast as they physically can (short-term decisions are more likely made at the other end, e.g. when to retire old systems, not add new ones). AWS was born out of the fact that Amazon's own compute needs are inherently variable so to meet peak demand they had to "over-provisio…

You're talking to someone who has spent the last decade working for major cloud providers, including AWS, on infrastructure and services sides of things, including work around data feeds for the capacity management teams. I have more than a passing familiarity with the way things actually work at a cloud.

They are constantly guessing at cloud capacity. Short, medium, and long term models with forecasting galore, all under constant recalculation based on customer actions (they literally take live feeds of creation/termination actions), and yes they also take in to account hardware failure and repair rates. Consolidating racks of equipment is a pain in the neck and tends to be avoided, unless you can safely live migrate away all instances.

They all build up various models, using all sorts of forecasting techniques. The longer range forecasts are involved in data center provisioning, along with other business analysis, market research, legal analysis etc. that helps define where future regions should be.

It's still a guess. They can't tell what the actual demand will be, and they can't tell what is going to happen with the supply chain (supply chain issues are the biggest nightmare for capacity planning teams). Sometimes they get it wrong.

The capacity management teams spend a lot of time and expertise to keep the company just sufficiently ahead of demand. It's a crucial part of keeping costs under control.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#292

Post by pamelafox »

websap wrote: post #110
bradknowles wrote: post #93

Earlier quoted context omitted.

Read the "Well Architected" paper. Go multi-region.

Can you link me to the well architected paper that talks about going multi-region for an early stage startup?

This may be the reference: https://learn.microsoft.com/en-us/azure/architecture/framewo...

I just watched the second video on the page, and it does discuss multi-region a bit.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#293

Post by count »

vageli wrote: post #231
count wrote: post #174

Earlier quoted context omitted.

In the AWS context, they are, in fact. That's the original point of them - so during big AZ failures your reserved instances had first dibs on the available capacity. The billing thing became more of the point as big AZ failures are so rare.

Non-zonal instance reservations do not reserve capacity. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-capa...

Well, yeah. But the zonal ones still do.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#294

Post by fock »

janober wrote: post #169
alexeldeib wrote: post #132

Earlier quoted context omitted.

Ah, bummer. If it helps, you can try this to list out VM sizes with comparable capabilities and see if you have better luck with any others (--all not really necessary since it filters by NotAvailableForSubscription and similar): az vm list-skus -l germanynorth -r virtualMachines --all > germanynorth.json jq '.[] | select( any( .capabilities[]; (.name == "vCPUs" and (.value | tonumber) >= 4 )) and any(.capabilities[]…

Thanks a lot! Just checked internally. Apparently are there some instances which we could get but would not work cost wise (have for example a lot of CPUs but we mainly care about RAM). Additionally, is there also still a region-wide CPU limit that would still cause us problems. So sadly not a long-term solution. But thanks a lot!

looking at the time you seem to spend on this issue and the fact you're apparently only needing low double digits of those instances.

Are you really sure you shouldn't just buy a bunch of machines (500cores/2TiB go for ~60k€), throw them into a colo and then spend that time on actually doing stuff?

Re: Ask HN: Azure has run out of compute – anyone else affected?

#295

Post by nerdbert »

RajT88 wrote: post #147
andrewstuart wrote: post #27

Message to cloud providers: List what you do you have available so we can choose. Do not force users to randomly guess and be refused until eventually finding something available.

Imagine if they did this in realtime. There's already DDOS attacks happening which are abusing the cloud free trials at scale - this would give them another attack vector. I can see why they wouldn't want to do this.

It only has to be available to paying customers. Or even customers over a certain paid usage threshold.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#296

Post by Nemo_bis »

arcturus17 wrote: post #270
DannyBee wrote: post #135

Most of Europe expects the winter to be quite painful from a power perspective. It would not be surprising if cloud providers (major power users) are being asked to not increase (or even decrease) power usage. The timeframe they gave would match that kind of ask. I wonder whether you see the same behavior from other cloud providers there (ie if you ask them whether new capacity is available, what do they say)

> It would not be surprising if cloud providers (major power users) are being asked to not increase (or even decrease) power usage. I doubt it. It will be easier - and probably safer - to ask citizens and physical industry (eg, factories) to bear the brunt than to risk having problems in critical IT infrastructure. Ask people and factories to turn the heat 3 degrees down and the effects will be more or less predictab…

It's not about shutting down existing machines. The power grid operator might be less willing to approve upgrades to serve increases in capacity. (No idea whether that's the case.)

Re: Ask HN: Azure has run out of compute – anyone else affected?

#297

Post by Nemo_bis »

Nemo_bis wrote: post #296
arcturus17 wrote: post #270

Earlier quoted context omitted.

> It would not be surprising if cloud providers (major power users) are being asked to not increase (or even decrease) power usage. I doubt it. It will be easier - and probably safer - to ask citizens and physical industry (eg, factories) to bear the brunt than to risk having problems in critical IT infrastructure. Ask people and factories to turn the heat 3 degrees down and the effects will be more or less predictab…

It's not about shutting down existing machines. The power grid operator might be less willing to approve upgrades to serve increases in capacity. (No idea whether that's the case.)

Someone else confirmed my guess https://news.ycombinator.com/item?id=33744179

Re: Ask HN: Azure has run out of compute – anyone else affected?

#298

Post by avereveard »

andrewstuart wrote: post #27

Message to cloud providers: List what you do you have available so we can choose. Do not force users to randomly guess and be refused until eventually finding something available.

https://aws.amazon.com/ec2/spot/instance-advisor/

It's not the exact metric but you can find which have more availability without knowing the exact number (which is constantly changing anyway)

Re: Ask HN: Azure has run out of compute – anyone else affected?

#299

Post by hitpointdrew »

whoknew1122 wrote: post #227
Twirrim wrote: post #211

Earlier quoted context omitted.

It's worth pointing out that every cloud is the same when it comes to capacity / capacity risk. They all apply a lot of time and effort to figuring out the optimal amount of capacity to order based on track record of both customer demand and supply chain satisfaction. Too much capacity is money spent getting no return, up front capex, ongoing opex, physical space in facilities etc. On cloud scales (averaged out over…

It may be a risk borne by every cloud provider, but why does this only really happen to Microsoft among large providers? As far as chip shortages, it probably helps that Amazon makes its own chips. Microsoft could do the same rather than running out of capacity and blaming chip shortages. Microsoft had to know that at some point they were going to run out of capacity. They should've either did something about it or l…

> As far as chip shortages, it probably helps that Amazon makes its own chips.

IDK what chips you are talking about, all x86 (which I assume is most of their compute) is Intel or AMD. If they make their own they are only making the ARM instances.

Post reply on HN