11–20 of 61 posts
AWS and GCP are not full
Feels like an opportunity for building a service that uses AWS, GCP or Azure based on which is cheapest at that moment + which is not "full"... Unless that already exists.
That would explain why I have to wait sometimes ~1h until the Azure DevOps queued pipelines build starts.
> Capacity constraints due to increased demand stemming from the global health pandemic are causing pipeline delays when using our hosted pools. We are working on mitigations, but currently expect the issue to persist for at least the rest of 25-March peak hours. You can work around these issues by temporarily moving critical pipelines to self-hosted agents.
I think this introduces some interesting points to the DR and BCP conversation. Is it a safe bet that we can rely on the cloud to have capacity? Normally I wouldn't doubt it but in this sort of situation is becomes more likely they will be put under capacity stress. Will the cloud vendors learn and build slack in? I think they're very lean operations and maybe this kind of slack would damage the profitability too muc…
Then you're very misinformed.
As a cloud administrator, I see resource availability and account limits on a weekly basis going back years.
I tell people:
- to pre-provision at least some extra servers rather than wait for an autoscaling operation to fail.
- that new instance types often are rolled out gradually, and lead time is often 1 month in AWS
- that killing a 1000-node cluster then expecting to immediately rebuild it often doesn't work.
- for DR and BCP planning, each region (or AZ) should be able to handle enough load at all times in case one region (or AZ) is unavailable. I've never seen anybody do that, even after I told them, because cost.
My personal experience for our AWS CI infra that it's struggling more and more recently. Builds are slower on average than a couple of weeks ago. Maybe those VCPUs are not the same VCPUs as yesterday ;D.
This is a bit like the financial crisis in 2008. In 2008, the idea was that if you bundle up a large bunch of mortgages, then the bundle will have low risk because the chances of everything failing at the same time is low. The cloud is designed so that resource usage spikes of individual customers can always be served because one customer is very small compared to the whole infrastructure. However, in some cases, the…
I think this introduces some interesting points to the DR and BCP conversation. Is it a safe bet that we can rely on the cloud to have capacity? Normally I wouldn't doubt it but in this sort of situation is becomes more likely they will be put under capacity stress. Will the cloud vendors learn and build slack in? I think they're very lean operations and maybe this kind of slack would damage the profitability too muc…
This is a bit like the financial crisis in 2008. In 2008, the idea was that if you bundle up a large bunch of mortgages, then the bundle will have low risk because the chances of everything failing at the same time is low. The cloud is designed so that resource usage spikes of individual customers can always be served because one customer is very small compared to the whole infrastructure. However, in some cases, the…
That's really an interesting analogy. Of course all analogies are wrong BUT if AWS goes down, half the internet goes down (in terms of important services).
There is something satisfying about the bounds of the cloud metaphor being reached (a cloud can't fill the sky).