Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Azure has run out of compute – anyone else affected?

news.ycombinator.com

Post reply on HN
Print view View original

311–320 of 359 posts

Re: Ask HN: Azure has run out of compute – anyone else affected?

#311

Post by gerdesj »

Twirrim wrote: post #291
femto113 wrote: post #278

Earlier quoted context omitted.

Cloud providers aren't guessing at demand to plan capacity, they're literally building new data centers and then wheeling new racks into them as fast as they physically can (short-term decisions are more likely made at the other end, e.g. when to retire old systems, not add new ones). AWS was born out of the fact that Amazon's own compute needs are inherently variable so to meet peak demand they had to "over-provisio…

You're talking to someone who has spent the last decade working for major cloud providers, including AWS, on infrastructure and services sides of things, including work around data feeds for the capacity management teams. I have more than a passing familiarity with the way things actually work at a cloud. They are constantly guessing at cloud capacity. Short, medium, and long term models with forecasting galore, all…

It's logistics no more and no less. Logistics has been a thing for ever (satisfy a resource requirement). My old man (is not a dustman) but he was Commander Supply for quite a lot of people. At one job, he and his staff would worry about things like Austrian plain chocolate covered mint centred frogs (I'm not joking) to Gurkha rice and not much else (some very concentrated protein etc) water-proofed combat rations. This was in Cyprus in the '80s. Logistics on the green line in Cyprus is probably still as mad now due to the number of countries in the UN.

Anyway, capacity planning is very well understood in general but of course the devil is in the details.

At the moment the IT supply chain is pretty spotty and that affects my little IT firm up to the big boys.

When you buy Cisco + HPE + Dell or whatevs, you go to your reseller (me). I go to my distributor and they suck hardware out of Dell etc and take their cut and I install the gear and take my cut. Sometimes a disty thinks they can do reseller too. The thinking is that they can roll up two lots of margin and shave a bit. That's fine if you can actually do logistics and the "teeth arm" job too.

Clouds think they can go even further and sometimes they can and sometimes not. Now we have a sodding complicated resource on offer with a supply chain that is a bit random.

The whole hyperscale cloud premise is based on infinite availability of raw resources and that is complete bollocks. You can't hyperscale if you can't source stuff indefinitely.

Those Austrian mint filled choccy frogs became a thing for a while. I gave no idea of the exact numbers but presumably Austria supplied quite a lot of them for the UN forces and families in Cyprus in the '80s - they became a bargaining chip for a while. They came in a cardboard package with a lid coloured light blue with outlines of frogs and I think the main box was dark brown or black.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#312

Post by dangus »

jiggawatts wrote: post #254
gumby wrote: post #184

Earlier quoted context omitted.

That pretty much binds your hands since in our experience the one provider who can do “one cloud with excellence” is AWS. (As an aside I also agree that multi cloud from the get go is a YAGNI violation. Just keep in the back of your mind “could we have an alternative to this?” when using your provider’s proprietary features.)

My experience is the opposite: AWS has more features on paper but most of them exist only to tick a checkbox. Azure has more integrations between their offerings, as well as Azure Active Directory, and Microsoft 365.

Why do I want AD or Microsoft 365?

Re: Ask HN: Azure has run out of compute – anyone else affected?

#313

Post by bane »

indoorskier wrote: post #277
marcinzm wrote: post #225

Earlier quoted context omitted.

In my experience there are differences between clouds so while all have the same basic problem in practice some may be better than others. I've never had issues getting GPUs on AWS but GCP constantly has issues with GPU/TPU capacity.

Is this region dependent? In us-east I can’t get them to approve a quota for GPU instance families (G,P) for anything more than 4 CPUs. At one point they rejected my request citing “unprecedented demand”. Of course this is small time, just my personal account. It is true I can get an instance most of the time, but not if I need >16GiB GPU memory.

We've been having the same problem getting GPU instances on us-east. Multiple week-long delays to escalate and talk to yet the next person up who can make a decision. It's a mess.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#314

Post by choward »

Ha. I knew something like this would happen eventually. Isn't limitless scalability one of the biggest selling points of using "the cloud"? If you have to buy your own computers anyway why even use the cloud? You could try using different clouds providers but eventually the clouds run out.

Which brings me to another important point. If we run out of computers meaning supply can't keep up with demand, then who are the winners? The people who own the computers. Cloud providers and self hosters. Because of the high demand cloud providers can raise their prices and that's directly converted to profit since expenses remain the same, i.e. price gouging. Good job all you cloud loyalists who use the cloud for everything.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#315

Post by Aeolun »

Twirrim wrote: post #260
whoknew1122 wrote: post #227

Earlier quoted context omitted.

It may be a risk borne by every cloud provider, but why does this only really happen to Microsoft among large providers? As far as chip shortages, it probably helps that Amazon makes its own chips. Microsoft could do the same rather than running out of capacity and blaming chip shortages. Microsoft had to know that at some point they were going to run out of capacity. They should've either did something about it or l…

There's all sorts of examples of AWS failing to be able to provide capacity too. Just do a search for "aws InsufficientInstanceCapacity" or similar. I remember Fortnite talking about capacity limits in relation to an incident, but I'm struggling to find the post-mortem I saw it in. Even when Microsoft was being open about Azure having difficulty getting Intel chips, AWS, GCP etc. were in the same position and just no…

It’s certainly true I run into these things with AWS as well, but it’s generally limited to a specific instance type/availability zone combination. I’ve never had all instance types unavailable.

If anything, I’m surprised we can just spin up a few hundred instances out of nowhere and not run into capacity issues.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#316

Post by philwelch »

cosmotic wrote: post #243
whoknew1122 wrote: post #227

Earlier quoted context omitted.

It may be a risk borne by every cloud provider, but why does this only really happen to Microsoft among large providers? As far as chip shortages, it probably helps that Amazon makes its own chips. Microsoft could do the same rather than running out of capacity and blaming chip shortages. Microsoft had to know that at some point they were going to run out of capacity. They should've either did something about it or l…

Amazon's own chips are ARM. ARM requires somewhat specialized builds of software that are likely different than development instances, CI/CD, and/or local dev machines. It's not insurmountable but does certainly complicate usage.

Your local dev machines might be Macs though, in which case it might be easier for you to go with ARM servers than x86.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#317

Post by SergeAx »

l-p wrote: post #131

> We never thought our startup would be threatened by the unreliability of a company like Microsoft You're new to Azure I guess. I'm glad the outage I had yesterday was only the third major one this year, though the one in august made me lose days of traffic, months of back and forth with their support, and a good chunk of my sanity and patience in face of blatant documented lies and general incompetence. One consume…

Yes, and what is your contingency plan for said fiber going dark?

Re: Ask HN: Azure has run out of compute – anyone else affected?

#318

Post by philwelch »

dangus wrote: post #312
jiggawatts wrote: post #254

Earlier quoted context omitted.

My experience is the opposite: AWS has more features on paper but most of them exist only to tick a checkbox. Azure has more integrations between their offerings, as well as Azure Active Directory, and Microsoft 365.

Why do I want AD or Microsoft 365?

You personally? No idea. You probably don't. But many (most?) businesses use AD and Office and aren't particularly interested in migrating to alternative solutions.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#319

Post by ckdarby »

l-p wrote: post #131

> We never thought our startup would be threatened by the unreliability of a company like Microsoft You're new to Azure I guess. I'm glad the outage I had yesterday was only the third major one this year, though the one in august made me lose days of traffic, months of back and forth with their support, and a good chunk of my sanity and patience in face of blatant documented lies and general incompetence. One consume…

> One consumer-grade fiber link is enough to serve my company's traffic and with two months of what we pay MS for their barely working cloud I could buy enough hardware to host our product for a year of two of sustained growth.

I don't believe that is even remotely correct.

It isn't the pricing you should be worried about but the staffing, redundancy, and 24/7 operations staff.

I'm dealing with AWS and on-prem. On-prem spent some $5M to build out a whole new setup, took literal multiple months of racking, planning, designing, setting up, etc.

It's not even entirely in use because we got supply chain issued for 100 Gbit switches and they won't be coming until at least April of 2023 (after many months of delays upon delays already).

Post reply on HN