Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Azure has run out of compute – anyone else affected?

news.ycombinator.com

Post reply on HN
Print view View original

301–310 of 359 posts

Re: Ask HN: Azure has run out of compute – anyone else affected?

#301

Post by arghwhat »

astrostl wrote: post #175
xwowsersx wrote: post #10

Oof, that sucks and I feel for you. That said... > setting up in a new region would be complicated for us. Sounds to me like you've got a few weeks to get this working. Deprioritize all other work, get everyone working on this little DevOps/Infra project. You should've been multi-region from the outset, if not multi-cloud. When using the public cloud, we do tend to take it all for granted and don't even think about t…

"Multi-cloud from the outset" is probably the single-worst generic cloud advice that I think anyone could be given. In professional cloud consulting the rule of thumb is to do one cloud with excellence until you even think about another one. And even that is really just kicking the conversational can, as both becoming excellent and actually needing multi-cloud combined is one-in-a-billion.

You always only need another one when everything has gone to shit, either from failure or cost from vendor-lock-in, so drinking your chosen providers Kool aid equals taking the reactive route and scrambling to rearchitect when the issues hit.

Multi-cloud is really not a big deal. Main nuisance is billing differences, followed by slight variations in e.g. Terraform config.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#302

Post by gumby »

jiggawatts wrote: post #254
gumby wrote: post #184

Earlier quoted context omitted.

That pretty much binds your hands since in our experience the one provider who can do “one cloud with excellence” is AWS. (As an aside I also agree that multi cloud from the get go is a YAGNI violation. Just keep in the back of your mind “could we have an alternative to this?” when using your provider’s proprietary features.)

My experience is the opposite: AWS has more features on paper but most of them exist only to tick a checkbox. Azure has more integrations between their offerings, as well as Azure Active Directory, and Microsoft 365.

We’ve had reliability and availability problems esp with azure and also Google. less with AWS.

None are ideal.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#303

Post by whoknew1122 »

Spooky23 wrote: post #288
whoknew1122 wrote: post #227

Earlier quoted context omitted.

It may be a risk borne by every cloud provider, but why does this only really happen to Microsoft among large providers? As far as chip shortages, it probably helps that Amazon makes its own chips. Microsoft could do the same rather than running out of capacity and blaming chip shortages. Microsoft had to know that at some point they were going to run out of capacity. They should've either did something about it or l…

They have a lot of technical debt. They have like 6 different clouds (at least 4 gov clouds alone) and SLA commitments to things like O365 that silo their infrastructure. MS also makes all sorts of crazy deals and commitments, and I wouldn’t be surprised if being collocated with a strategic customer may lead to local shortages of resources.

AWS has at least 3 publicly-discussed 'clouds' (or partitions, as they're called at AWS). There may or may not be other partitions that cannot be discussed publicly.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#304

Post by arcturus17 »

Nemo_bis wrote: post #296
arcturus17 wrote: post #270

Earlier quoted context omitted.

> It would not be surprising if cloud providers (major power users) are being asked to not increase (or even decrease) power usage. I doubt it. It will be easier - and probably safer - to ask citizens and physical industry (eg, factories) to bear the brunt than to risk having problems in critical IT infrastructure. Ask people and factories to turn the heat 3 degrees down and the effects will be more or less predictab…

It's not about shutting down existing machines. The power grid operator might be less willing to approve upgrades to serve increases in capacity. (No idea whether that's the case.)

That makes more sense.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#305

Post by whoknew1122 »

hitpointdrew wrote: post #299
whoknew1122 wrote: post #227

Earlier quoted context omitted.

It may be a risk borne by every cloud provider, but why does this only really happen to Microsoft among large providers? As far as chip shortages, it probably helps that Amazon makes its own chips. Microsoft could do the same rather than running out of capacity and blaming chip shortages. Microsoft had to know that at some point they were going to run out of capacity. They should've either did something about it or l…

> As far as chip shortages, it probably helps that Amazon makes its own chips. IDK what chips you are talking about, all x86 (which I assume is most of their compute) is Intel or AMD. If they make their own they are only making the ARM instances.

AWS has three processors: Graviton, Inferentia, and Trainium. They're made in-house.

https://aws.amazon.com/silicon-innovation/

Re: Ask HN: Azure has run out of compute – anyone else affected?

#306

Post by theknocker »

astrostl wrote: post #175
xwowsersx wrote: post #10

Oof, that sucks and I feel for you. That said... > setting up in a new region would be complicated for us. Sounds to me like you've got a few weeks to get this working. Deprioritize all other work, get everyone working on this little DevOps/Infra project. You should've been multi-region from the outset, if not multi-cloud. When using the public cloud, we do tend to take it all for granted and don't even think about t…

"Multi-cloud from the outset" is probably the single-worst generic cloud advice that I think anyone could be given. In professional cloud consulting the rule of thumb is to do one cloud with excellence until you even think about another one. And even that is really just kicking the conversational can, as both becoming excellent and actually needing multi-cloud combined is one-in-a-billion.

No post body was provided.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#307

Post by jepler »

adrr wrote: post #195
l-p wrote: post #131

> We never thought our startup would be threatened by the unreliability of a company like Microsoft You're new to Azure I guess. I'm glad the outage I had yesterday was only the third major one this year, though the one in august made me lose days of traffic, months of back and forth with their support, and a good chunk of my sanity and patience in face of blatant documented lies and general incompetence. One consume…

Azure has some of the biggest outages like when they went down on Feb29th for the whole day. https://azure.microsoft.com/en-us/blog/summary-of-windows-az...

It seems like in nearly 3 out of every 4 years the whole internet is unusable on February 29... why pick on microsoft?

Re: Ask HN: Azure has run out of compute – anyone else affected?

#308

Post by discordance »

andrewstuart wrote: post #69
arecurrence wrote: post #58

This is not as rare as public clouds may lead people to believe. I have had to move workloads around since AWS began (even between public clouds on occasion). In particular, GPU availability has been a continuing problem. Unlike interchangeable x64 / arm64 instances with some adjustments based on the new core and ram count... if no GPU instances are available then I simply cannot run the job. AMD's improved support h…

GPUs are better run in your own office. All cloud providers charge much, much more for GPUs than if you run a local machine. Cloud GPUs are also a lot slower than state of the art consumer GPUs. Cloud GPUs: much slower, less available, much more expensive.

For small orgs this makes sense, but this really depends on how big your data sets are that you're training against and how your ML Ops / Data Ops is set up.

GPUs are better run close to your data. If you're training on-prem then your data needs to be on-prem too.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#309

Post by AmericanChopper »

marcinzm wrote: post #225
Twirrim wrote: post #211

Earlier quoted context omitted.

It's worth pointing out that every cloud is the same when it comes to capacity / capacity risk. They all apply a lot of time and effort to figuring out the optimal amount of capacity to order based on track record of both customer demand and supply chain satisfaction. Too much capacity is money spent getting no return, up front capex, ongoing opex, physical space in facilities etc. On cloud scales (averaged out over…

In my experience there are differences between clouds so while all have the same basic problem in practice some may be better than others. I've never had issues getting GPUs on AWS but GCP constantly has issues with GPU/TPU capacity.

I’ve occasionally seen some of the internal AWS capacity management dashboards, and they can frequently be operating very close to 100% on some resource types.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#310

Post by analyst74 »

DannyBee wrote: post #135

Most of Europe expects the winter to be quite painful from a power perspective. It would not be surprising if cloud providers (major power users) are being asked to not increase (or even decrease) power usage. The timeframe they gave would match that kind of ask. I wonder whether you see the same behavior from other cloud providers there (ie if you ask them whether new capacity is available, what do they say)

Obviously Azure failed its customer here, but everyone with data centers in Europe is tightening their belts and preparing for the worst.

I suspect AWS and GCP just have more headroom in EU.

Post reply on HN