Tell HN: AWS appears to be down again • Hacker News Board

Re: Tell HN: AWS appears to be down again

#381

Post by electroly » Wed, Dec 22, 2021, 3:45 PM UTC

pm90 wrote: post #294

spmurrayzzz wrote: post #256

Earlier quoted context omitted.

Your last comment is really important, I think. I have always petitioned for "passive over active" design in distributed cloud systems. The recent outages, and also ones from the past, demonstrate why. The fewer API calls you need to make in-band with whatever throughput is generated via your customer demand, the better. Related to that, I have been critical of lambda/FaaS/serverless infrastructure patterns for simil…

> The fewer API calls you need to make in-band with whatever throughput is generated via your customer demand, the better. Agreed; however, this is somewhat difficult to do correctly. There are all sorts of systems that might have hidden dependencies on managed services. e.g. AWS IAM roles will almost always be checked at some point if your services need to interact with AWS managed services. I think cloud providers…

AWS IAM is designed with a control plane / data plane dichotomy. Even if the control plane is completely dead and all API requests are failing, services in a steady-state (i.e. not responding to changes via API calls) can still rely on IAM roles using their cached information. For example, in the recent us-east-1 outage when you couldn't start new services because IAM checks would fail, existing EC2 instances that rely on IAM instance profiles to access services like S3 could still do so even though IAM was down.

Re: Tell HN: AWS appears to be down again

#382

Post by StreamBright » Wed, Dec 22, 2021, 3:45 PM UTC

lmilcin wrote: post #275

quantumfissure wrote: post #261

Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…

Think about it this way: 1) Can you make your on prem infrastructure go down less than Amazon's? 2) Is it worth it? In my experience most people grossly underestimate how expensive it is to create reliable infrastructure and at the same time overestimate how important it is for their services to run uninterrupted. -- EDIT: I am not arguing you shouldn't build your more reliable infrastructure. AWS is just a point on…

3) Could you hire talent that can build the thing?

In my experience problem number 3 is the hardest to solve.

Re: Tell HN: AWS appears to be down again

#383

Post by NovemberWhiskey » Wed, Dec 22, 2021, 3:45 PM UTC

jsmith99 wrote: post #340

NovemberWhiskey wrote: post #314

Earlier quoted context omitted.

I think it's predicated on a misunderstanding of what "fail-safe" actually means. For example, in railway signaling, drivers are trained to interpret a signal with no light as the most restrictive aspect (e.g. "danger"). That way, any failure of a bulb in a colored light signal, or a failure of the signal as a whole, results in a safe outcome (albeit that the train might be delayed while the driver calls up the signa…

Or nuclear reactors that fail safe by dropping all the control rods into the core to stop all activity. The reactor may be permanently ruined after that (with a cost of hundreds of millions or billions to revert) but there will be no risk of meltdown.

I don't know enough about reactor control systems to be sure on that one. The idea of a fail-safe system is not that there's an easy way to shut them down, but more that the ways we expect the component parts of a system to fail result in the safe state.

e.g. consider a railway track circuit - this is the way that a signaling system knows whether a particular block of a track is occupied by a train or not. The wheels and axle are conductive so you can measure this electrically by determining whether there's a circuit between the rails or not.

The naive way to do this would be to say something like "OK, we'll apply a voltage to one rail, and if we see a current flowing between the rails we'll say the block is occupied." This is not fail-safe. Say the rail has a small break, or if power is interrupted: no current will flow, so the track always looks unoccupied even if there's a train.

The better way is to say "We'll apply a voltage to one rail, but we'll have the rails connected together in a circuit during normal operation. That will energize a relay which will cause the track to indicate clear. If a train is on the track, then we'll get a short circuit, which will cause the relay to de-energize, indicating the track is occupied."

If the power fails, it shows the track occupied because the relay opens. If the rail develops a crack, the circuit opens, again causing the relay to open and indicate the track is occupied. If the relay fails, then as long as it fails open (which is the predominant failure mode of relays) the track is also indicated as occupied.

Re: Tell HN: AWS appears to be down again

#384

Post by dylan604 » Wed, Dec 22, 2021, 3:45 PM UTC

tluyben2 wrote: post #361

JCM9 wrote: post #303

AWS didn’t “go down”. They had an outage in one AZ, which is why there are multiple AZs in each region. If your app went down then you should be blaming your developers on this one, not AWS. Those having issues are discovering gaps in their HA designs. Obviously it’s not good for an AZ to go down but it does happen and why any production workload should be architected to have seamless failover and recover to other AZ…

> People commenting that servers shouldn’t go down ect don’t understand how true HA architectures work. You should expect and build for stuff to fail like this. Otherwise it’s like complaining that you lost data because a disk failed. Disks fail… build architecture where that won’t take you down. Is that comparison fair? If you have 2 raid-5 mirrored raid 5 boxes in your room and all disks fail at the same time, you…

>And that won't happen

HA! I had received new 16-bay chasis and all of the drives needed plus cold spares for each chasis. Set them up and started the RAID-5 init on a Friday. Left them running in the rack over the weekend. Returned on Monday to find multiple drives in each chasis had failed. Even with dedicated one of the 16 drives as a hot swap, the volumes would all have failed in an unrecoverable manner.

All drives were purchased at the same time, and happened to all come from a single batch from the manufacture. The manufacture confirmed this via serial numbers, and admitted they had an issue during production. All drives were replaced and at a larger volume size.

TL;DR: Drives will fail, and manufacturing issues happend. Don't buy all of your drives in an array from the same batch! It will happen. To say it won't is just pure inexeperience.

Re: Tell HN: AWS appears to be down again

#385

Post by truetraveller » Wed, Dec 22, 2021, 3:46 PM UTC

BiteCode_dev wrote: post #351

osrec wrote: post #262

Earlier quoted context omitted.

Purely out of interest, I'd like to know more about your streaming architecture. I assume postgres just holds the meta data, and the actual video content is stored elsewhere? What strategies have you employed to scale the streaming part of your service? I imagine 4 million users a month is quite a significant amount of traffic!

1 - For the last 10 years, servers have been beasts. You have a lot of cores, plenty of HD and RAM. Servers are less expensives than devs. Scaling vertically can go VERY far. 2 - Caching is life. We have 3 layers of caching: cloudflare, varnish, and redis. Most things don't need to be real time. A lot of things can be a month old and the user doesn't care. User need immediate feedback to be happy, but not necessary f…

Thank you!

Re: Tell HN: AWS appears to be down again

#386

Post by meshaneian » Wed, Dec 22, 2021, 3:46 PM UTC

joana035 wrote: post #367

uvdn7 wrote: post #331

Earlier quoted context omitted.

You could have just showed them historical data of both companies being unavailable for extended amount of time. What happened in the past few months is not new.

"just", as if you never had to argument against aws fanboys...

As a pragmatic AWS fan, +1 this. Disposable distributed hybrid multi-cloud architecture FTW.

Re: Tell HN: AWS appears to be down again

#387

Post by vel0city » Wed, Dec 22, 2021, 3:46 PM UTC

dijit wrote: post #362

praseodym wrote: post #359

Earlier quoted context omitted.

OP is talking about the DC power feed, not a single server PSU.

You don't get fed DC power, you get fed AC power. But, point taken: yes your power feed should be running at Mostly this is outsourced to the datacenter provider; they'll give you a per side rating. (usually 10A or 16A) which also matches the cooling profile of the cabinet.

I mean, in some datacenters they run DC power to each rack. Its definitely more esoteric than having each device run AC but some people do it.

However, with their comment DC == Data Center, not Direct Current.

Re: Tell HN: AWS appears to be down again

#388

Post by manquer » Wed, Dec 22, 2021, 3:47 PM UTC

kalleth wrote: post #375

manquer wrote: post #342

Earlier quoted context omitted.

S3 and (others) have version history that can be enabled. If you have to take care of availablity and redundancy and delete protection and backups then why pay the premium S3 is charging ? Either you don't trust the cloud and you can run NAS or equivalent (with s3 APIs easily today) much cheaper or trust them to keep your data safe and available. No point in investing in S3 and then doing it again yourself.

In most startups? You're mostly correct. But you still have some risks here, yes, with a super low probability, but a company-killing impact. In some industries - banking, finance, anything regulated, or really (I'd argue) anywhere where losing all of your data is company killing - you will need a disaster recovery strategy in place. The risks requiring non-AWS backups are things like: - A failed payment goes unnotic…

Yes threat models is obvious qualifier, if you have a business that requires backup on the moon if there asteroid collision then by all means got for it.[1]

For most companies what AWS.or Azure offers is more than adequate.

An internal bad actor with that level of privileged access can delete your local backups or external one can all things you he can do to AWS he can likely do easier to your company storage DC too.

Bottom-line it doesn't matter if customers can pay for all this low probability stuff that can only happen on the cloud and not on Prem sure go ahead. Half the things customers pay for they don't need or use anyway.

[1] assuming your business model allows you to spend the expense outlay you need for the threat model

Re: Tell HN: AWS appears to be down again

#389

Post by dijit » Wed, Dec 22, 2021, 3:47 PM UTC

vel0city wrote: post #380

dijit wrote: post #302

Earlier quoted context omitted.

EDIT: I misunderstood you were talking about power feeds, the normal case is the run "48% as if it's 100%" (because of power spikes, but also most types of transformers run more efficiently under specific levels of load (40-60). Normally this is factored into the Rack you buy from a hardware provider, they will tell you that you have 10A or 16A on each feed, if you exceed that: it will work, but you are overloading t…

The poster was speaking more of the power delivery going to the power supplies, not the server's power supplies themselves. So say each PSU 1 is wired to circuit A, each PSU 2 is wired to circuit B. Circuit A experiences a failure. All servers instantly switch over all their load to their PSU 2's on circuit B. Suddenly circuit B's load is roughly double what it was just moments ago. If proper planning wasn't created…

Yeah I understand on re-reading: but that's also not how people run datacenters.

Obviously people can operate things however they want, but you wont get a tier 3 classification with that setup.

Re: Tell HN: AWS appears to be down again

#390

Post by phamilton » Wed, Dec 22, 2021, 3:47 PM UTC

boudin wrote: post #357

JCM9 wrote: post #303

AWS didn’t “go down”. They had an outage in one AZ, which is why there are multiple AZs in each region. If your app went down then you should be blaming your developers on this one, not AWS. Those having issues are discovering gaps in their HA designs. Obviously it’s not good for an AZ to go down but it does happen and why any production workload should be architected to have seamless failover and recover to other AZ…

Issues are across all us-east 1, not one AZ. Load balancers are not doing well at all. The only way in this case to avoid an outage is to be cross regions or cross cloud which is quite more complex to handle and require more resources to do well. And I hope that nobody is listening your blaming and pointing fingers advice, that's the worst way to solve anything. It's AWS job to ensure that things are reliable, that t…

Echoing this. We had to manually intervene and cut off the faulty AZ because our ASGs kept spinning up instances in it and our load balancers kept sending traffic to bad hosts.

In the past I've seen both of those systems seamlessly handle an AZ failure. Today was different.