Tell HN: Azure outage • Hacker News Board

Re: Tell HN: Azure outage

#151

Post by NetMageSCW » Wed, Oct 29, 2025, 5:02 PM UTC

ocdtrekkie wrote: post #101

kierenj wrote: post #18

Ouch, and login.microsoftonline.com too - i.e. SSO using MS accounts. We'd just rolled that out across most (all?) of our internal systems... And microsoft.com too - that's gotta hurt

I am still stunned people choose to do this, considering major Office 365 outages are basically a weekly thing now.

We are very dependent on Azure and Microsoft Authentication and Microsoft 365 and haven’t had weekly or even monthly issues. I can think of maybe three issues this year.

Re: Tell HN: Azure outage

#152

Post by chuckadams » Wed, Oct 29, 2025, 5:02 PM UTC

speckx wrote: post #137

FYI: https://status.cloud.microsoft/

Which itself is^H^H was down. Wow.

Re: Tell HN: Azure outage

#153

Post by thimkerbell » Wed, Oct 29, 2025, 5:03 PM UTC

Does (should, could) DownDetector also say what customer-facing services are down, when some infrastructure is unworking? Or is that the info that the malefactors are seeking?

Re: Tell HN: Azure outage

#154

Post by drewnick » Wed, Oct 29, 2025, 5:03 PM UTC

ape4 wrote: post #62

2026: the year of your own metal in a rack

I've been doing it since 1998 in my bedroom with a dual T1 (and on to real DCs later). While I've had some outages for sure it makes me feel better I am not that divergent in uptime in the long run vs big clouds.

Re: Tell HN: Azure outage

#155

Post by Mr_Bees69 » Wed, Oct 29, 2025, 5:03 PM UTC

kryogen1c wrote: post #27

downdetector reports coincident cloudflare outage. is microsoft using cloudflare for management plane, or is there common infra? data center problem somewhere, maybe fiber backbone? BGP?

nope, dont see any cf issues.

Re: Tell HN: Azure outage

#156

Post by [deleted] » Wed, Oct 29, 2025, 5:03 PM UTC

[deleted]

Re: Tell HN: Azure outage

#157

Post by nartaczact » Wed, Oct 29, 2025, 5:03 PM UTC

djeastm wrote: post #48

I'm mid-deployment, but thankfully it seems to be running ok so far. Just the portal is not working so my visibility is not good.

Sounds like Shrodinger's Deploy

Re: Tell HN: Azure outage

#158

Post by computerdork » Wed, Oct 29, 2025, 5:04 PM UTC

withinboredom wrote: post #129

SoftTalker wrote: post #112

Earlier quoted context omitted.

Mind-boggling that any retailer would not have the capability to at least run the checkout stations offline.

I knew an old guy in the '00s who specialized in cobal/fortran for working on tiller software. Guess he retired and they couldn't maintain it

Anyone remember Bob's number?? Bob?! Oh the humanity! We're all gonna be canned!

Re: Tell HN: Azure outage

#159

Post by parliament32 » Wed, Oct 29, 2025, 5:04 PM UTC

Insanity wrote: post #128

8cvor6j844qw_d6 wrote: post #78

Earlier quoted context omitted.

I'll be interested in the incident writeup since DNS is mentioned. It will be interesting in a way if it is similar to what happened at AWS.

It's pretty unlikely. AWS published a public 'RCA' https://aws.amazon.com/message/101925/ . A race condition in a DNS 'record allocator' causing all DNS records for DDB to be wiped out. I'm simplifying a bit, but I don't think it's likely that Azure has a similar race condition wiping out DNS records on _one_ system than then propagates to all others. The similarity might just end at "it was DNS".

That RCA was fun. A distributed system with members that don't know about each other, don't bother with leader elections, and basically all stomp all over each other updating the records. It "worked fine" until one of the members had slightly increased latency and everything cascade-failed down from there. I'm sure there was missing (internal) context but it did not sound like a well-architected system at all.

Re: Tell HN: Azure outage

#160

Post by chasd00 » Wed, Oct 29, 2025, 5:04 PM UTC

Jamie452 wrote: post #52

Currently standing in a half closed supermarket because the tills are down and they cant take payments

There's a Family Dollar by my house that is down at least 2 full days per month because of bad inet connectivity. I live close enough that with a small tower on my roof i can get line of sight to theirs. I've thought about offering them a backup link off my home inet if they give me 50% of sales whenever its in use. It would be a pretty good deal for them, better some sales when their inet is down vs none.