Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Azure outage

news.ycombinator.com

Post reply on HN
Print view View original

791–800 of 841 posts

Re: Tell HN: Azure outage

#791

Post by ape4 »

thayne wrote: post #789
xnorswap wrote: post #701

Earlier quoted context omitted.

33 minutes from impact to status page for a complete outage is a joke.

Unfortunately,that is also typical. I've seen it take longer than that for AWS to update their status page. The reason is probably because changes to the status page require executive approval, because false positives could lead to bad publicity, and potentially having to reimburse customers for failing to meet SLAs.

Perhaps they could set the time to when it really started after executive approval.

Re: Tell HN: Azure outage

#793

Post by oofbey »

croemer wrote: post #682

Preliminary post incident review: https://azure.status.microsoft/en-gb/status/history/ Timeline 15:45 UTC on 29 October 2025 – Customer impact began. 16:04 UTC on 29 October 2025 – Investigation commenced following monitoring alerts being triggered. 16:15 UTC on 29 October 2025 – We began the investigation and started to examine configuration changes within AFD. 16:18 UTC on 29 October 2025 – Initial communication po…

“Our protection mechanisms, to validate and block any erroneous deployments, failed due to a software defect which allowed the deployment to bypass safety validations.”

Very circular way of saying “the validator didn’t do its job”. This is AFAICT a pretty fundamental root cause of the issue.

It’s never good enough to have a validator check the content and hope that finds all the issues. Validators are great and can speed a lot of things up. But because they are independent code paths they will always miss something. For critical services you have to assume the validator will be wrong, and be prepared to contain the damage WHEN it is wrong.

Re: Tell HN: Azure outage

#794

Post by HeavyStorm »

xnorswap wrote: post #701
croemer wrote: post #682

Preliminary post incident review: https://azure.status.microsoft/en-gb/status/history/ Timeline 15:45 UTC on 29 October 2025 – Customer impact began. 16:04 UTC on 29 October 2025 – Investigation commenced following monitoring alerts being triggered. 16:15 UTC on 29 October 2025 – We began the investigation and started to examine configuration changes within AFD. 16:18 UTC on 29 October 2025 – Initial communication po…

33 minutes from impact to status page for a complete outage is a joke.

I've been on bridges where people _forgot_ to send comms for dozens of minutes. Too many inexperienced people around these days.

Re: Tell HN: Azure outage

#795

Post by WorldMaker »

xnorswap wrote: post #702
btown wrote: post #433

Earlier quoted context omitted.

Azure to me has always suffered from a belief that “UI innovations can solve UX complexity if you just try hard enough.” Like, AWS, and GCP to a lesser extent, has a principled approach where simple click-ops goals are simple. You can access the richer metadata/IAM object model at any time, but the wizards you see are dumb enough to make easy things easy. With Azure, those blades allow tremendously complex “you need…

I still feel lost just trying to view my application logs. I don't want to pay for or lock myself into, "Azure Insights". I just want to see the logging, that I know if I can remember the right buttons to click, are available. The worst place to try is "Monitoring > Logs", this is where you get faced up front with a query designer. I've never worked out how to do a simple "list by time" on that query designer, but it…

A lot of Azure Insights is a value add on top of OTel. You can use "just" OTel feeding Azure Insights (and that's what the modern Aspire-influenced defaults mostly do) and possibly avoid that feeling of vendor lockin. That perspective might also give you ideas of other "modern" vendors to audition such as Grafana if you wanted to see what other people are doing with OTel rather than Event Logs and file system logs.

Re: Tell HN: Azure outage

#796

Post by lokar »

sfn42 wrote: post #737
neya wrote: post #708

Earlier quoted context omitted.

In Microsoft's defense, Azure has always been a complete joke. It's extremely developer unfriendly, buggy and overpriced.

I've only used Azure, to me it seems fine ish. Some things are rather overcomplicated and it's far from perfect but I assumed the other providers were similarly complicated and imperfect. Can't say I've experienced many bugs in there either. It definitely is overpriced but I assume they all are?

For something fairly small, they are about the same.

At a large scale, azure is dramatically worse then Aws.

Re: Tell HN: Azure outage

#797

Post by taco_emoji »

onionisafruit wrote: post #687
croemer wrote: post #682

Preliminary post incident review: https://azure.status.microsoft/en-gb/status/history/ Timeline 15:45 UTC on 29 October 2025 – Customer impact began. 16:04 UTC on 29 October 2025 – Investigation commenced following monitoring alerts being triggered. 16:15 UTC on 29 October 2025 – We began the investigation and started to examine configuration changes within AFD. 16:18 UTC on 29 October 2025 – Initial communication po…

At 16:04 “Investigation commenced”. Then at 16:15 “We began the investigation”. Which is it?

    16:04 Started running around screaming
    16:15 Sat down & looked at logs

Re: Tell HN: Azure outage

#798

Post by notorandit »

croemer wrote: post #682

Preliminary post incident review: https://azure.status.microsoft/en-gb/status/history/ Timeline 15:45 UTC on 29 October 2025 – Customer impact began. 16:04 UTC on 29 October 2025 – Investigation commenced following monitoring alerts being triggered. 16:15 UTC on 29 October 2025 – We began the investigation and started to examine configuration changes within AFD. 16:18 UTC on 29 October 2025 – Initial communication po…

What puzzles me too is the time it took to recognize an outage.

Looks like there was no monitoring and no alerts.

Which is kinda weird.

Re: Tell HN: Azure outage

#799

Post by ukblewis »

voidfunc wrote: post #767
sfn42 wrote: post #737

Earlier quoted context omitted.

I've only used Azure, to me it seems fine ish. Some things are rather overcomplicated and it's far from perfect but I assumed the other providers were similarly complicated and imperfect. Can't say I've experienced many bugs in there either. It definitely is overpriced but I assume they all are?

They are all broken, weird, and expensive in their own ways. Its nothing unique to Azure.

Some are much worse than others…

Re: Tell HN: Azure outage

#800

Post by sfn42 »

lokar wrote: post #796
sfn42 wrote: post #737

Earlier quoted context omitted.

I've only used Azure, to me it seems fine ish. Some things are rather overcomplicated and it's far from perfect but I assumed the other providers were similarly complicated and imperfect. Can't say I've experienced many bugs in there either. It definitely is overpriced but I assume they all are?

For something fairly small, they are about the same. At a large scale, azure is dramatically worse then Aws.

Worse at what?
Post reply on HN