Live data from Hacker News

Ask HN: Azure has run out of compute – anyone else affected?

news.ycombinator.com

211–220 of 359 posts

Re: Ask HN: Azure has run out of compute – anyone else affected?

#211
post #131

> We never thought our startup would be threatened by the unreliability of a company like Microsoft You're new to Azure I guess. I'm glad the outage I had yesterday was only the third major one this year, though the one in august made me lose days of traffic, months of back and forth with their support, and a good chunk of my sanity and patience in face of blatant documented lies and general incompetence. One consume…

It's worth pointing out that every cloud is the same when it comes to capacity / capacity risk. They all apply a lot of time and effort to figuring out the optimal amount of capacity to order based on track record of both customer demand and supply chain satisfaction.

Too much capacity is money spent getting no return, up front capex, ongoing opex, physical space in facilities etc.

On cloud scales (averaged out over all the customers) the demand tends to follow pretty stable and predictable patterns, and the ones that actually tend to put capacity at risk (large customers) have contracts where they'll give plenty of heads-up to the providers.

What has been very problematical over the past few years has been the supply chains. Intel's issues for a few years in getting CPUs out really hurt the supply chains. All of the major providers struggled through it, and the market is still somewhat unpredictable. The supply chain woes that have been wrecking chaos with everything from the car industry to the domestic white goods industry are having similar impacts on the server industry.

The level of unreliability in the supply chain is making it very difficult for the capacity management folks to do their job. It's not even that predictable which supply chain is going to be affected. Some of them are running far smoother and faster and capacity lands far faster than you'd expect, while others are completely messed up, then next month it's all flipped around. They're being paranoid, assuming the worst and still not getting it right.

This is an area where buying physical hardware directly doesn't provide any particular advantages. Their supply chains are just as messed up.

The best thing to try to do is do your best to be as hardware agnostic as is technically possible, so you can use whatever is available... which sucks.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#212
post #131

> We never thought our startup would be threatened by the unreliability of a company like Microsoft You're new to Azure I guess. I'm glad the outage I had yesterday was only the third major one this year, though the one in august made me lose days of traffic, months of back and forth with their support, and a good chunk of my sanity and patience in face of blatant documented lies and general incompetence. One consume…

The common argument of "our own hardware would be more profitable in X years" is typically countered with "but you need to pay engineers to maintain it, which adds to the cost". Another advantage of not having to own the hardware is that it's easier to scale, and get started with new types of services. (i.e, datawarehouse solutions, serverless compute, new DB types,..). I'm not trying to advocate for or against cloud…

Depends on how stable your needs are, but sometimes its cheaper even when you considerer total cost and not just for big deployments.

In the past 2 or three years, we probably moved more services off the cloud than other way. That said one reason for that is that most new services are build in the cloud, so there are less services off the cloud than on it.

Cloud is best, when you are starting out, when you don't know what you need, need high velocity of adding new stuff, of have very burst like demand for either traffic or cpu etc. Or if you are just small developer only team.

But if you have applications that are relatively stable, are mostly feature complete and you don't expect much sudden growth etc, it's useful to run the numbers if cloud is still something you want/need.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#213
post #64

Earlier quoted context omitted.

Seems bold to recommend everyone do the same as you when you are running in to problems you can't solve because of this exact choice you made.

I am still 100% sure it was exactly the right decision. Was however in hindsight probably the right one to choose Azure and/or that data center.

If you go under because of this, will you still be 100% sure?

Everything is for sure until it’s not.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#215
post #24

Earlier quoted context omitted.

This is an insidious argument to make. When building a startup you should choose 1 reliable cloud provider and use their best practices to support high availability.

No matter the provider, their best practices all say to be multi-region.

Multi-region in AWS means building it yourself.

I suspect that the skills for real HA are atrophying because for 99% of the people multi-AZ is enough and most of the AWS stuff supports multi-az automagically.

The problem with multi-region is that it means configuration, and there are probably lots of services that you can't actually configure to be multi-region. Cognito is one off the top of my head. It looks like the various aurora flavors do multi-region, but what about Neptune? SQS? API Gateway? AWS Lambda? MediaLive?

Maybe you can hide all that behind DNS failover, maybe you can't.

Real multi-region is basically means going back to old-school HA, and that was hard to do when it was your data centers. On AWS it'll be even harder.

That isn't to say it's not possible, it's just a tremendous amount of work.

I mean really, if us-east-1 is down 80% of the internet is screwed...so from an expectations point of view does HA of your particular service matter if that happens? Even for a financial outages happen.

Once you have enough people it might be worth it. For a non mission critical startup? No fucking way.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#216
post #143

Get in touch with your CSAM. They will be able to get you assigned a capacity manager, if you don't already have one assigned. It is the function of the capacity manager to help you plan ahead based on what the data center capacities look like going into the future. Meet monthly with your capacity manager. Get representation across different technology interests - database, compute, storage, event hubs, etc. Don't ev…

Not much better than "meet with Infrastructure in Nov to plan next years capacity and server purchases" for on prep -- has Azure really degraded down to this?

It's quite a bit better than that, in fact. They talk to their customers to try and understand all the big deployments coming to understand if there is going to be a crunch at the region/AZ level.

I'd be surprised if other cloud providers aren't doing that in some form. I only have experience with Azure (so far).

Re: Ask HN: Azure has run out of compute – anyone else affected?

#217
post #83

Earlier quoted context omitted.

No matter the provider, their best practices all say to be multi-region.

Multi-AZ and then grow into multi-region if the need arises. Multi-region is a huge lift the moment all your data must live in two regions simultaneously. Very few shops are experienced enough to run clusters across datacenters in a way that can handle the unhappy paths.

[deleted]

Re: Ask HN: Azure has run out of compute – anyone else affected?

#218
post #180

Earlier quoted context omitted.

I take offense with you comment. It's not the first time I'm hearing about multi-region/multi-cloud in online tech forums, however reality doesn't match. I don't want to be snarky, but when large service providers like AWS have their own crossregion downtime because one snowflake of a service in us-east-1 is down, I kind of dismiss the virtue signaling of high resilient multi-(az/region/cloud) ever existing in practi…

> dismiss the virtue signaling of high resilient ... So you're saying it's impossible to improve reliability from 97% to 99% because you can never make it to 100%.

If your single-AZ, single-region cloud is not giving you 3 or 4 9's of reliability out of the box, you are using the wrong cloud.

Multi-AZ and multi-region add complexity and cost much more quickly than they add reliability.

Sometimes it is worth it. Sometimes it is not.

Re: Ask HN: Azure has run out of compute – anyone else affected?

#219
post #131

> We never thought our startup would be threatened by the unreliability of a company like Microsoft You're new to Azure I guess. I'm glad the outage I had yesterday was only the third major one this year, though the one in august made me lose days of traffic, months of back and forth with their support, and a good chunk of my sanity and patience in face of blatant documented lies and general incompetence. One consume…

Out of curiosity (from someone inexperienced with Azure), is it a skill/ability chasm between MS engineering and outsourced support?

TAMs tend to be a bandaid organizational sign that support-as-normal sucks and isn't sufficient to get the job done (ie fix everything that breaks and isn't self-serve).

Re: Ask HN: Azure has run out of compute – anyone else affected?

#220
>These problems seem only in the German region, but setting up in a new region would be complicated for us.

This seems like your fundamental problem. If you design an architecture that is limited to a single region of a single cloud provider, you are very likely to encounter issues at some point.

Luckily you have a full month to solve this problem before it will prevent you from accepting new users. My suggestion is to start making your app multi-regional or multi-provider ASAP.

Post reply on HN