Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Azure outage

news.ycombinator.com

Post reply on HN
Print view View original

601–610 of 841 posts

Re: Tell HN: Azure outage

#601

Post by nijave »

RajT88 wrote: post #231
cyptus wrote: post #224

Earlier quoted context omitted.

AFD is down quite often regionally in Europe for our services. In 50%+ the cases they just don‘t report it anywhere, even if its for 2h+.

Spam those Azure tickets. If you have a CSAM, build them a nice powerpoint telling the story of all your AFD issues (that's what they are there for). > In 50%+ the cases they just don‘t report it anywhere, even if its for 2h+. I assume you mean publicly. Are you getting the service health alerts?

Back when we used Azure the only outcome was them trying to upsell us on Premium Support

Re: Tell HN: Azure outage

#602

Post by bigstrat2003 »

hedayet wrote: post #348

The sad thing is - $MSFT isn't even down by 1%. And IIRC, $AMZN actually went up during their previous outage. So if we look at these companies' bottom lines, all those big wigs are actually doing something right. Sales and lobbying capacity is way more effective than reliability or good engineering (at least in the short term).

That's a good thing. Stock prices shouldn't go down because of rare incidents which don't accurately represent how successful a company is likely to be in the future.

Re: Tell HN: Azure outage

#603

Post by ceejayoz »

esseph wrote: post #593
vasco wrote: post #562

Earlier quoted context omitted.

They both share the fact that you don't see your vote enter a ballot box.

Seeing your ballot drop in a box is no indicator the vote is actually recorded in the grand tally, or what was recorded for your vote.

My county lets you look up if it was received. You can vote on Election Day in person if they don’t.

Re: Tell HN: Azure outage

#604

Post by nijave »

parliament32 wrote: post #159
Insanity wrote: post #128

Earlier quoted context omitted.

It's pretty unlikely. AWS published a public 'RCA' https://aws.amazon.com/message/101925/ . A race condition in a DNS 'record allocator' causing all DNS records for DDB to be wiped out. I'm simplifying a bit, but I don't think it's likely that Azure has a similar race condition wiping out DNS records on _one_ system than then propagates to all others. The similarity might just end at "it was DNS".

That RCA was fun. A distributed system with members that don't know about each other, don't bother with leader elections, and basically all stomp all over each other updating the records. It "worked fine" until one of the members had slightly increased latency and everything cascade-failed down from there. I'm sure there was missing (internal) context but it did not sound like a well-architected system at all.

>slightly increased latency

They didn't provide any details on latency. It could have been delayed an hour or a day and no one noticed

Re: Tell HN: Azure outage

#605

Post by maxdamantus »

trevoragilbert wrote: post #591
arsome wrote: post #587

Earlier quoted context omitted.

One of these things is much easier to burn or otherwise tamper with.

You should research what’s inside the boxes in Oregon before just assuming they’re easier to tamper with.

Doesn't look difficult: https://www.fbi.gov/wanted/seeking-info/ballot-box-fires (yes, that's in Oregon)

Re: Tell HN: Azure outage

#606

Post by dotancohen »

stefs wrote: post #565
foresterre wrote: post #464

It still surprises me how much essential services like public transport are completely reliant on cloud providers, and don't seem to have backups in place. Here in The Netherlands, almost all trains were first delayed significantly, and then cancelled for a few hours because of this, which had real impact because today is also the day we got to vote for the next parlement (I know some who can't get home in time befor…

i'm not sure this is an easily solvable problem. i remember reading an article arguing that your cloud provider is part of your tech stack and it's close to impossible/a huge PITA to make a non-trivial service provider-agnostic. they'd have to run their own openstack in different datacenters, which would be costly and have their own points of failure.

I run non trivial services on EC2, using that service as a VPS. My deploy script works just as well on provisioned Digital Ocean services and on docker containers using docker-compose.

I do need a human to provision a few servers and configure e.g. load balancing and when to spin up additional servers under load. But that is far less of a PITA than having my systems tied to a specific provider or down whenever a cloud precipitates.

Re: Tell HN: Azure outage

#607

Post by progmetaldev »

onionisafruit wrote: post #417
david422 wrote: post #172

Earlier quoted context omitted.

IIRC, the grocery chain I worked for used to have an offline mode to move customers out the door. But it meant that when the system came back online, if the customers card was denied, the customer got free groceries.

What I gather from this is to always try a dead card first just in case the store is in offline mode

They still capture the name on the card, so I would be careful about trying this, unless you can make use of a prepaid card.

Re: Tell HN: Azure outage

#608

Post by baby_souffle »

dotancohen wrote: post #606
stefs wrote: post #565

Earlier quoted context omitted.

i'm not sure this is an easily solvable problem. i remember reading an article arguing that your cloud provider is part of your tech stack and it's close to impossible/a huge PITA to make a non-trivial service provider-agnostic. they'd have to run their own openstack in different datacenters, which would be costly and have their own points of failure.

I run non trivial services on EC2, using that service as a VPS. My deploy script works just as well on provisioned Digital Ocean services and on docker containers using docker-compose. I do need a human to provision a few servers and configure e.g. load balancing and when to spin up additional servers under load. But that is far less of a PITA than having my systems tied to a specific provider or down whenever a clou…

It's absolutely doable if you design for it.

The moment you choose to use S3 instead of hosting your own object store, though, you either use AWS because S3 and IAM already have you or spend more time on the care and feeding of your storage system as opposed to actually doing the thing you customers are paying you to do.

It's not impossible, just complicated and difficult for any moderately complex architecture.

Re: Tell HN: Azure outage

#609

Post by dotancohen »

RajT88 wrote: post #408
cyberax wrote: post #333

Earlier quoted context omitted.

> CSAM Child Sex-Abuse Material?!? Well, a nice case of acronym collision.

Definitely the most baffling acronym collision I have seen with Microsoft. I did one time count 4 different products abbreviated VSTS at one point.

Didn't MS have three things called "link" at one time? They were all spelled differently, of course.

Re: Tell HN: Azure outage

#610

Post by Noumenon72 »

RajT88 wrote: post #405
riffic wrote: post #383

Earlier quoted context omitted.

In general, plain language works so much better than throwing bowls of alphabet soup around.

That's a funny criticism to make on a tech forum. But, for future reference: site:microsoft.com csam

That's an even 5:5 split between both meanings.
Post reply on HN