Earlier quoted context omitted.
Or maybe we should throw them in jail.
I agree, but because the coffee is crap
And ridiculously expensive
Tell HN: Azure outage
711–720 of 841 posts
Re: Tell HN: Azure outage
#712Preliminary post incident review: https://azure.status.microsoft/en-gb/status/history/ Timeline 15:45 UTC on 29 October 2025 – Customer impact began. 16:04 UTC on 29 October 2025 – Investigation commenced following monitoring alerts being triggered. 16:15 UTC on 29 October 2025 – We began the investigation and started to examine configuration changes within AFD. 16:18 UTC on 29 October 2025 – Initial communication po…
33 minutes from impact to status page for a complete outage is a joke.
and for a while the status was "there might be issues on azure portal".
Re: Tell HN: Azure outage
#713Earlier quoted context omitted.
33 minutes from impact to status page for a complete outage is a joke.
In Microsoft's defense, Azure has always been a complete joke. It's extremely developer unfriendly, buggy and overpriced.
If you call that defending microsoft, I'd hate to see what attacking them looks like :)
Re: Tell HN: Azure outage
#714Preliminary post incident review: https://azure.status.microsoft/en-gb/status/history/ Timeline 15:45 UTC on 29 October 2025 – Customer impact began. 16:04 UTC on 29 October 2025 – Investigation commenced following monitoring alerts being triggered. 16:15 UTC on 29 October 2025 – We began the investigation and started to examine configuration changes within AFD. 16:18 UTC on 29 October 2025 – Initial communication po…
33 minutes from impact to status page for a complete outage is a joke.
More importantly
`15:45 UTC on 29 October 2025 – Customer impact began.
16:04 UTC on 29 October 2025 – Investigation commenced following monitoring alerts being triggered. ` A 19-minute delay in alert is a joke.
Re: Tell HN: Azure outage
#715Somewhere, an ex microsoft engineer that where layoff during the last week, is saying to himself “thank god, this shit is not my problem anymore”
Re: Tell HN: Azure outage
#716For some reason an Azure outage does not faze me in the same way that an AWS outage does. I have never had much confidence in Azure as a cloud provider. The vertical integration of all the things for a Microsoft shop was initially very compelling. I was ready to fight that battle. But, this fantasy was quickly ruined by poor execution on Microsoft's part. They were able to convince me to move back to AWS by simply ma…
I like azure. I think they are more intuitive tua aws and good, and they have good prices for startups ( essentially free for a whole year )
Re: Tell HN: Azure outage
#717Looking forward to the post mortem.
> What went wrong and why?
> An inadvertent tenant configuration change within Azure Front Door (AFD) triggered a widespread service disruption affecting both Microsoft services and customer applications dependent on AFD for global content delivery. The change introduced an invalid or inconsistent configuration state that caused a significant number of AFD nodes to fail to load properly, leading to increased latencies, timeouts, and connection errors for downstream services.
> As unhealthy nodes dropped out of the global pool, traffic distribution across healthy nodes became imbalanced, amplifying the impact and causing intermittent availability even for regions that were partially healthy. We immediately blocked all further configuration changes to prevent additional propagation of the faulty state and began deploying a ‘last known good’ configuration across the global fleet. Recovery required reloading configurations across a large number of nodes and rebalancing traffic gradually to avoid overload conditions as nodes returned to service. This deliberate, phased recovery was necessary to stabilize the system while restoring scale and ensuring no recurrence of the issue.
> The trigger was traced to a faulty tenant configuration deployment process. Our protection mechanisms, to validate and block any erroneous deployments, failed due to a software defect which allowed the deployment to bypass safety validations. Safeguards have since been reviewed and additional validation and rollback controls have been immediately implemented to prevent similar issues in the future.
So, so far they're saying it's a combination of bad config + their config-validator had a bug. Would love more details.
Re: Tell HN: Azure outage
#718It still surprises me how much essential services like public transport are completely reliant on cloud providers, and don't seem to have backups in place. Here in The Netherlands, almost all trains were first delayed significantly, and then cancelled for a few hours because of this, which had real impact because today is also the day we got to vote for the next parlement (I know some who can't get home in time befor…
Could you provide a reference about the train disruption? I tried but couldn't find anything in English.
Re: Tell HN: Azure outage
#719Earlier quoted context omitted.
Have repeated outages lost them customers? has it lost them any money in any way? That is a pass.
Apologies, but this just reads like a low effort critique of big things. To be clear, they should get criticism. They should be held liable for any damage they cause. But that they remain the biggest cloud offering out there isn't something you'd expect to change from a few outages that, by most all evidence, potential replacements have, as well? More, a lot of the outages potential replacements have are often more g…
I would say you are explaining why they get a free pass so they still get one - they are bad but their main competitors are even worse!
I thought one of the major selling points of the big cloud providers was that they were more reliable than running your own stuff (by which i mean anything from a VPS to multiple data centres depending on your scale. Compared to those alternatives they seem to be less reliable in practice!
The solution is to have a multi-region, or even multi-cloud setup, but then bang goes the "they do all the work for you" argument (which i doubt anyway).
Re: Tell HN: Azure outage
#720Earlier quoted context omitted.
33 minutes from impact to status page for a complete outage is a joke.
and for a while the status was "there might be issues on azure portal".
There might have been, but they didn't know because they couldn't access it. Could have been something totally unrelated.