Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Azure has run out of compute – anyone else affected?

news.ycombinator.com

Post reply on HN
Print view View original

281–290 of 359 posts

Re: Ask HN: Azure has run out of compute – anyone else affected?

#281

Post by deathanatos »

gtirloni wrote: post #62
andrewstuart wrote: post #47

Earlier quoted context omitted.

No, this is my actual experience using azure.

I can go on Azure right now and create an instance and nobody will check anything manually and email me back something. Maybe you're confusing Azure with some other small town colocation provider.

Accounts have instance count quotas; you can get them raised, but it is a support ticket to do so.

And sometimes, that is hard. I've had Azure support not able to understand what quota they need to raise / what quota is being requested. I had to at least link them to their own documentation on it… (partly the confusion is that quota support tickets allow selecting the quota as a piece of metadata on the ticket, but only for some quotas, and of course, mine was for one of the ones not listed. Why they don't just list all of them is anyone's guess.)

Re: Ask HN: Azure has run out of compute – anyone else affected?

#282

Post by deathanatos »

I've seen this before. I think it was in us-west1, ran out of VMs of the size we used for CI. Had to move to a different region. (Never moved back…)

It is shocking to me that it happened at all. Capacity planning shouldn't be so far behind in a cloud that wants to position it as being on-par with AWS/GCP. (Which Azure absolutely isn't.) To me, having capacity planning be solved is part of what I am paying for in that higher price of the VM.

> We never thought our startup would be threatened by the unreliability of a company like Microsoft, or that they wouldn’t proactively inform us about this.

Oh my sweet summer child, welcome to Azure. Don't depend on them being proactive about anything; even depending on them to react is a mistake, e.g., they do not reliably post-mortem severe failures. (At least, externally. But as a customer, I want to know what you're doing to prevent $massive_failure from happening again, and time and time again they're just silent on that front.)

Re: Ask HN: Azure has run out of compute – anyone else affected?

#283

Post by bushbaba »

macrolime wrote: post #236
cosmic_quanta wrote: post #125

Earlier quoted context omitted.

I would argue that the cost profile is different. Local GPUs are a big up-front cost. But assuming that your workload is stable, in the long run I think local GPUs ends up being cheaper per-hour than cloud. For startups, it doesn't make sense to make the up-front purchase, fine. But if you're optimizing for long-term (amortized) costs, I'd be curious if cloud is cost-effective.

The long run being three months

An nvidia dgx box is roughly 40k. And that’s not including power/storage/rack Space.

But yes, If a single workstation can meet your gpu training needs then it’ll be cheaper with sufficient usage

Re: Ask HN: Azure has run out of compute – anyone else affected?

#284

Post by Spooky23 »

Twirrim wrote: post #211
l-p wrote: post #131

> We never thought our startup would be threatened by the unreliability of a company like Microsoft You're new to Azure I guess. I'm glad the outage I had yesterday was only the third major one this year, though the one in august made me lose days of traffic, months of back and forth with their support, and a good chunk of my sanity and patience in face of blatant documented lies and general incompetence. One consume…

It's worth pointing out that every cloud is the same when it comes to capacity / capacity risk. They all apply a lot of time and effort to figuring out the optimal amount of capacity to order based on track record of both customer demand and supply chain satisfaction. Too much capacity is money spent getting no return, up front capex, ongoing opex, physical space in facilities etc. On cloud scales (averaged out over…

> This is an area where buying physical hardware directly doesn't provide any particular advantages. Their supply chains are just as messed up.

Yup. And a few of the OEMs have stopped talking about supply chain integrity. Many folks have observed more memory and power supply problems since the pandemic.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#286

Post by remus »

unionpivo wrote: post #229
Innominate wrote: post #121

Earlier quoted context omitted.

As much as this happens, I don't feel it's something to be expected or even okay. The major cloud services are expensive. This extra cost is supposed to provide for cloud services' high level of flexibility. Running out of capacity should be a rare event and treated as a high priority problem to be fixed asap. Without the ability to rapidly and arbitrarily scale, they're just overpriced server farms.

> Without the ability to rapidly and arbitrarily scale, they're just overpriced server farms. I mean that's what cloud is (outsourced server farm). Sure they also offer services on top, but that's mostly because they want to lock you in, and can charge more for, so it's a win win for them. And there is no magic here, someone has to get the chips, build servers and connect them to network. And while they will often ov…

> I don't think any cloud will ever be able to guarantee to never run out of resources.

I agree with this, but clearly there's a disconnect between how often people expect these kinds of issues and how often they actually happen. The whole point of the cloud is you pay a premium for the added flexibility. If it turns out that flexibility isn't there when you need it then maintaining your own servers becomes a lot more attractive.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#287

Post by janober »

robjan wrote: post #118

We've been having this problem in Singapore for a couple of years now. Can't add any VMs to our k8s cluster and can't provision a number of services which made our multi-region BCP more complicated.

Years?!?! Guess I then have to be happy that in our case it is "just" around 4 months.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#288

Post by Spooky23 »

whoknew1122 wrote: post #227
Twirrim wrote: post #211

Earlier quoted context omitted.

It's worth pointing out that every cloud is the same when it comes to capacity / capacity risk. They all apply a lot of time and effort to figuring out the optimal amount of capacity to order based on track record of both customer demand and supply chain satisfaction. Too much capacity is money spent getting no return, up front capex, ongoing opex, physical space in facilities etc. On cloud scales (averaged out over…

It may be a risk borne by every cloud provider, but why does this only really happen to Microsoft among large providers? As far as chip shortages, it probably helps that Amazon makes its own chips. Microsoft could do the same rather than running out of capacity and blaming chip shortages. Microsoft had to know that at some point they were going to run out of capacity. They should've either did something about it or l…

They have a lot of technical debt. They have like 6 different clouds (at least 4 gov clouds alone) and SLA commitments to things like O365 that silo their infrastructure.

MS also makes all sorts of crazy deals and commitments, and I wouldn’t be surprised if being collocated with a strategic customer may lead to local shortages of resources.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#289

Post by Spooky23 »

ethbr0 wrote: post #219
l-p wrote: post #131

> We never thought our startup would be threatened by the unreliability of a company like Microsoft You're new to Azure I guess. I'm glad the outage I had yesterday was only the third major one this year, though the one in august made me lose days of traffic, months of back and forth with their support, and a good chunk of my sanity and patience in face of blatant documented lies and general incompetence. One consume…

Out of curiosity (from someone inexperienced with Azure), is it a skill/ability chasm between MS engineering and outsourced support? TAMs tend to be a bandaid organizational sign that support-as-normal sucks and isn't sufficient to get the job done (ie fix everything that breaks and isn't self-serve).

Microsoft support is really awful. Basically, if you need it regularly, you just pay for resident engineers who can bypass the wall between the product groups and you. I’ve had nothing but great experiences with those guys.

Otherwise, especially if there’s a broader problem, they play lots of games with SLAs, etc.

Post reply on HN