Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Azure outage

news.ycombinator.com

Post reply on HN
Print view View original

821–830 of 841 posts

Re: Tell HN: Azure outage

#821

Post by hinkley »

notorandit wrote: post #798
croemer wrote: post #682

Preliminary post incident review: https://azure.status.microsoft/en-gb/status/history/ Timeline 15:45 UTC on 29 October 2025 – Customer impact began. 16:04 UTC on 29 October 2025 – Investigation commenced following monitoring alerts being triggered. 16:15 UTC on 29 October 2025 – We began the investigation and started to examine configuration changes within AFD. 16:18 UTC on 29 October 2025 – Initial communication po…

What puzzles me too is the time it took to recognize an outage. Looks like there was no monitoring and no alerts. Which is kinda weird.

I've seen sensitivity get tuned down to avoid false positives during deployments or rolling restarts for host updates. And to a lesser extent for autoscaling noise. It can be hard to get right.

I think it's perhaps a gap in the tools. We apply the same alert criteria at 2 am that we do while someone is actively running deployment or admin tasks and there's a subset that should stay the same, like request failure rate, and others that should be tuned down, like overall error rate and median response times.

And it means one thing if the failure rate for one machine is 90% and something else if the cluster failure rate is 5%, but if you've only got 18 boxes it's hard to discern the difference. And which is the higher priority error may change from one project to another.

Re: Tell HN: Azure outage

#822

Post by tremon »

hinkley wrote: post #818
infaloda wrote: post #714

Earlier quoted context omitted.

More importantly `15:45 UTC on 29 October 2025 – Customer impact began. 16:04 UTC on 29 October 2025 – Investigation commenced following monitoring alerts being triggered. ` A 19-minute delay in alert is a joke.

10 minutes to alert, to avoid flapping false positives. 10 minute response window for first responders. Or, 5 minute window before failing over to backup alerts, and 4 minutes to wake up, have coffee, and open the appropriate windows.

I'd like to think that a company the size of Microsoft can afford to have incident response teams in enough time zones to cover basic operations without relying on night shifts.

Re: Tell HN: Azure outage

#823

Post by hinkley »

tremon wrote: post #822
hinkley wrote: post #818

Earlier quoted context omitted.

10 minutes to alert, to avoid flapping false positives. 10 minute response window for first responders. Or, 5 minute window before failing over to backup alerts, and 4 minutes to wake up, have coffee, and open the appropriate windows.

I'd like to think that a company the size of Microsoft can afford to have incident response teams in enough time zones to cover basic operations without relying on night shifts.

That’s some very carefully chosen phrasing.

I think if you really wanted to do on call right to avoid gaps you’d want no more than 6 hours on primary per day per shift, and you want six, not four, shifts per day. So you’re only alone for four hours in the middle of your shift and have plenty of time to hand off.

Re: Tell HN: Azure outage

#824

Post by Tostino »

dotancohen wrote: post #811
Tostino wrote: post #779

Earlier quoted context omitted.

There are plenty of compatible S3-like offerings. That's one of the lesser things that tie me to a cloud.

Even on non-AWS projects, I still use S3. I haven't really explored the other options, but if you have opinions or advice I'd love to hear them. One thing very important, is that I can authorise specific web clients (users) to access specific resources from S3. Such as a document that he can download, but others with the link should not be able to download. Thank you!

The way I solved auth in my case was just proxying everything through my backend and having that do the auth. I have my own URL scheme and the users never see the URL for the file in S3.

Another way you can do it is generating pre-signed URLs in your backend on each request to download something... but the URL that is generated when you do that is only valid for some small time period, so not a stable URL at all.

In my use case, I needed stable URLs, so I went the proxy route.

Re: Tell HN: Azure outage

#825

Post by deadbolt »

notorandit wrote: post #798
croemer wrote: post #682

Preliminary post incident review: https://azure.status.microsoft/en-gb/status/history/ Timeline 15:45 UTC on 29 October 2025 – Customer impact began. 16:04 UTC on 29 October 2025 – Investigation commenced following monitoring alerts being triggered. 16:15 UTC on 29 October 2025 – We began the investigation and started to examine configuration changes within AFD. 16:18 UTC on 29 October 2025 – Initial communication po…

What puzzles me too is the time it took to recognize an outage. Looks like there was no monitoring and no alerts. Which is kinda weird.

Just what you want in a cloud provider, right?

Re: Tell HN: Azure outage

#826

Post by lawgimenez »

ac2u wrote: post #771
lawgimenez wrote: post #766

Earlier quoted context omitted.

Sounds like he’s describing Windows phone.

Feel like I have to defend windows phone here, I liked it! Although I swore off the platform after the hardware I bought wasn’t eligible for the windows phone 8 upgrade even though the hardware was less than two years old. They punished early adopters

Yeah Windows Phone's first releases were decent. I have developed apps for Windows actually using Window's UWP framework but there weren't enough users on their platform sadly.

Re: Tell HN: Azure outage

#827

Post by the_black_hand »

multiplegeorges wrote: post #445
bob1029 wrote: post #362

For some reason an Azure outage does not faze me in the same way that an AWS outage does. I have never had much confidence in Azure as a cloud provider. The vertical integration of all the things for a Microsoft shop was initially very compelling. I was ready to fight that battle. But, this fantasy was quickly ruined by poor execution on Microsoft's part. They were able to convince me to move back to AWS by simply ma…

> At this point I'd rather use GCP over Azure and I have zero seconds of experience with it. TBH, GCP is very good! More people should use it.

have used both. I prefer Azure. Much easier to use. found GCP unintuitive.

Re: Tell HN: Azure outage

#829

Post by GuestFAUniverse »

alias_neo wrote: post #736
dude250711 wrote: post #727

Earlier quoted context omitted.

Save it for when they stick Copilot into Azure portal.

Ha, you haven't used it recently have you? Copilot is already there, and it can't do a single useful thing. Me: "How do I connect [X] to [Y] using [Z]?" Copilot: "Please select the AKS cluster you'd like to delete"

Perfect answer /s

Re: Tell HN: Azure outage

#830

Post by neop1x »

croemer wrote: post #682

Preliminary post incident review: https://azure.status.microsoft/en-gb/status/history/ Timeline 15:45 UTC on 29 October 2025 – Customer impact began. 16:04 UTC on 29 October 2025 – Investigation commenced following monitoring alerts being triggered. 16:15 UTC on 29 October 2025 – We began the investigation and started to examine configuration changes within AFD. 16:18 UTC on 29 October 2025 – Initial communication po…

>> We began the investigation and started to examine configuration changes within AFD.

Troubleshooting has completed

Troubleshooting was unable to automatically fix all of the issues found. You can find more details below.

>> We initiated the deployment of our ‘last known good’ configuration.

System Restore can help fix problems that might be making your computer run slowly or stop responding.

System Restore does not affect any of your documents, pictures, or other personal data. Recently installed programs and drivers might be uninstalled.

Confirm your restore point

Your computer will be restored to the state it was in before the event in the Description field below.

Post reply on HN