Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Azure has run out of compute – anyone else affected?

news.ycombinator.com

Post reply on HN
Print view View original

251–260 of 359 posts

Re: Ask HN: Azure has run out of compute – anyone else affected?

#251

Post by Godel_unicode »

adrr wrote: post #195
l-p wrote: post #131

> We never thought our startup would be threatened by the unreliability of a company like Microsoft You're new to Azure I guess. I'm glad the outage I had yesterday was only the third major one this year, though the one in august made me lose days of traffic, months of back and forth with their support, and a good chunk of my sanity and patience in face of blatant documented lies and general incompetence. One consume…

Azure has some of the biggest outages like when they went down on Feb29th for the whole day. https://azure.microsoft.com/en-us/blog/summary-of-windows-az...

10 years ago, has there been something similar recently?

Re: Ask HN: Azure has run out of compute – anyone else affected?

#252

Post by jnsaff2 »

janober wrote: post #20
xwowsersx wrote: post #10

Oof, that sucks and I feel for you. That said... > setting up in a new region would be complicated for us. Sounds to me like you've got a few weeks to get this working. Deprioritize all other work, get everyone working on this little DevOps/Infra project. You should've been multi-region from the outset, if not multi-cloud. When using the public cloud, we do tend to take it all for granted and don't even think about t…

Thanks a lot! You are totally right, it is for sure something we will find a solution for. But honestly, do I not want to. As a startup, you have very few resources and deliberately place some exact bets. Deprioritizing everything to work on something for a long time that was not prioritized, just to then end up again where you were before (a working cloud solution) is the last thing any startup should be forced to d…

I feel for you. Also it sucks to be in this position.

Let the scar you get from this is be a learning experience, hopefully you will not fall into the same trap again to trust this company.

In my career I'm in a place where anyone suggesting I do work on Azure gets an instant doubling of my asking day-rate and I really hope the will be put off and find another victim for this gig.

That said, another learning experience would be to use terraform or something (tbh for azure the only sane thing is terraform, ARM templates are just garbage). Having terraformed your one region switching to the other would be much easier, tho not trivial.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#253

Post by jiggawatts »

more_corn wrote: post #238
Twirrim wrote: post #211

Earlier quoted context omitted.

It's worth pointing out that every cloud is the same when it comes to capacity / capacity risk. They all apply a lot of time and effort to figuring out the optimal amount of capacity to order based on track record of both customer demand and supply chain satisfaction. Too much capacity is money spent getting no return, up front capex, ongoing opex, physical space in facilities etc. On cloud scales (averaged out over…

All cloud providers are NOT equal here. Amazon over-provisions and sells the excess capacity as spot instances.

So does Azure.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#254

Post by jiggawatts »

gumby wrote: post #184
astrostl wrote: post #175

Earlier quoted context omitted.

"Multi-cloud from the outset" is probably the single-worst generic cloud advice that I think anyone could be given. In professional cloud consulting the rule of thumb is to do one cloud with excellence until you even think about another one. And even that is really just kicking the conversational can, as both becoming excellent and actually needing multi-cloud combined is one-in-a-billion.

That pretty much binds your hands since in our experience the one provider who can do “one cloud with excellence” is AWS. (As an aside I also agree that multi cloud from the get go is a YAGNI violation. Just keep in the back of your mind “could we have an alternative to this?” when using your provider’s proprietary features.)

My experience is the opposite: AWS has more features on paper but most of them exist only to tick a checkbox. Azure has more integrations between their offerings, as well as Azure Active Directory, and Microsoft 365.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#256

Post by 29athrowaway »

jiggawatts wrote: post #254
gumby wrote: post #184

Earlier quoted context omitted.

That pretty much binds your hands since in our experience the one provider who can do “one cloud with excellence” is AWS. (As an aside I also agree that multi cloud from the get go is a YAGNI violation. Just keep in the back of your mind “could we have an alternative to this?” when using your provider’s proprietary features.)

My experience is the opposite: AWS has more features on paper but most of them exist only to tick a checkbox. Azure has more integrations between their offerings, as well as Azure Active Directory, and Microsoft 365.

And Active Directory integrates horribly with everything outside Microsoft.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#258

Post by dilyevsky »

pclmulqdq wrote: post #80
jakear wrote: post #49

People saying "shame on you for not being multi-region" are missing the point: This is a German company with German customers subject to German data residency laws. For them to store German data in a region besides Germany requires getting informed consent from the "data subject", who must be "pre-informed about the potential risks involved in cross-border data transfer". [1] This is why Azure has a dedicated German…

Time to sign up for AWS or GCP, then. If you're using kubernetes anyway, you'll be fine with the switch.

I’ve had capacity issues before with both gcp and aws in smaller regions so not a panacea

Re: Ask HN: Azure has run out of compute – anyone else affected?

#259

Post by mmcconnell1618 »

wstuartcl wrote: post #183
mmcconnell1618 wrote: post #81

I used to be a technical seller for Azure. This situation is obviously not great for you as a customer but there are proactive steps you can take to prevent this going forward. Reach out to your sales team and work with them on your roadmap for compute requirements going forward. The sales team has a forecast tool that feeds back into the department that buys and racks the equipment. If you can provide enough lead ti…

What you describe is like the inverse of 90% of the reason companies host in the cloud. What makes needing to forcast and reach out to a sales guy to eventually stock hardware for your needs (while now competing against other customers for those resources) any better than hosting on prem. AWS for sure has had resource constraints in different AZs (especially during flack Friday and holiday loads) but I have never had…

Under most circumstance, this isn't needed unless you have a big ask. Say, you need 1,000 specific cores and GPUs, then this process is the best way to ensure you have them available.

The original poster probably has the ability to spin up other instance types in their region. If there is no compute capacity in the entire region, something went wrong operationally.

I'm not suggesting you should put in a request for every new resource you need, but if you have a specific instance type or a large number needed, it helps. You're not losing the ability to shut them down the next day if you don't need them, you're just telling the Azure team that you expect to spin some up around a certain time. If you're making a significant request of compute capacity, the team has the ability to reserve those instances for your subscriptions so that you're not competing with others for those cores.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#260

Post by Twirrim »

whoknew1122 wrote: post #227
Twirrim wrote: post #211

Earlier quoted context omitted.

It's worth pointing out that every cloud is the same when it comes to capacity / capacity risk. They all apply a lot of time and effort to figuring out the optimal amount of capacity to order based on track record of both customer demand and supply chain satisfaction. Too much capacity is money spent getting no return, up front capex, ongoing opex, physical space in facilities etc. On cloud scales (averaged out over…

It may be a risk borne by every cloud provider, but why does this only really happen to Microsoft among large providers? As far as chip shortages, it probably helps that Amazon makes its own chips. Microsoft could do the same rather than running out of capacity and blaming chip shortages. Microsoft had to know that at some point they were going to run out of capacity. They should've either did something about it or l…

There's all sorts of examples of AWS failing to be able to provide capacity too. Just do a search for "aws InsufficientInstanceCapacity" or similar. I remember Fortnite talking about capacity limits in relation to an incident, but I'm struggling to find the post-mortem I saw it in.

Even when Microsoft was being open about Azure having difficulty getting Intel chips, AWS, GCP etc. were in the same position and just not really talking about it. From my time in AWS there were some other times when some services with specialised hardware came really, really close to running out of capacity and had to scramble around with major internal "fire drills" against services to recoup capacity.

Most people won't run in to these issues, the clouds all tend to be good at it, but they still happen.

There are also advantages of the economy of scale and brand recognition. The more customers you have the more the capacity trends smooth out, the easier it is to predict need, even if you're still stuck with uncertainty on the ordering side.

Post reply on HN