No post body was provided.
Tell HN: AWS appears to be down again
581–590 of 646 posts
Re: Tell HN: AWS appears to be down again
#582Earlier quoted context omitted.
Self-managed infrastructure doesn’t fail now?
What an absolutely pointless comment. Everything fails, we can argue the rate. But I would argue that understanding your constraints is better. if you know that your secret storage system can't survive if a machine goes away: well, you wire redundant paths to the hardware and do memory mirroring and RAID the hell out of the disks. And if it fails you have a standby in place. But if you use AWS Cognito. And it goes do…
If you think you can do better than AWS, GCP, Azure there is a lot of money to be made, for sure.
Re: Tell HN: AWS appears to be down again
#583Earlier quoted context omitted.
This is quite interesting as they claim their datacenter design does better than Uptime's Tier3+ design requirements which require redundant power supply paths. [ https://aws.amazon.com/compliance/uptimeinstitute/ ]. I really hope they publish a thorough RCA for this incident.
"Electrical power systems are designed to be fully redundant so that in the event of a disruption, uninterruptible power supply units can be engaged for certain functions, while generators can provide backup power for the entire facility." https://aws.amazon.com/compliance/data-center/infrastructure... So they have 2 different sources of power coming in. And generators. They do mention the UPS is only for "certain fu…
Likely the UPS can't run HVAC, and you are in an overheat condition in about two minutes with a fully loaded data center without cooling. Proportionately longer as load is reduced.
Re: Tell HN: AWS appears to be down again
#584Earlier quoted context omitted.
Really? You'd think at Amazon's scale an additional PSU in a 1U custom-built server (I assume they're custom) would be a few tens of $ at most. Actually, now that I type that it makes sense. Scaling a few tens of dollars to a bajillion servers on the off-chance that you get an inbound power failure (quite rare I'd reckon) might cost more than what they'd lose if it does actually fail. So yeah, they're potentially jus…
At big cloud provider scale like Amazon, Azure, and Google they probably aren't even running PSUs at each server, they're probably doing DC at the rack these days. No point in having a million little transformers everywhere, far easier maintenance centralizing those and have multiple feeding the bus bars going to each rack.
The ones Im seeing designed have been moving the DC out to the cabinets with A/B 480VAC power feeds on the bus, and integrated DC inverters/rectifiers/batteries at the rack level.
More modular and a lot less copper at 10x the voltage. Still a lot of copper.
Re: Tell HN: AWS appears to be down again
#585If you haven't seen yet, news is it was a power loss: > 5:01 AM PST We can confirm a loss of power within a single data center within a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. This is affecting availability and connectivity to EC2 instances that are part of the affected data center within the affected Availability Zone. We are also experiencing elevated RunInstance API error rates for launches wi…
So dumb question from someone who hasn't maintained large public infrastructure: Isn't the whole point of availability zones is that you deploy to more than one and support failing over if one fails? IE why are we (consumers) hearing about this or being obviously impacted (eg Epic Games Store is very broken right now)? Is my assessment wrong, or are all these apps that are failing built wrong? Or something in between…
Amazon shifts the responsibility for multi-AZ deployment to us customers, saving themselves complexity and charging us extra - win-win for them.
Re: Tell HN: AWS appears to be down again
#586I've built out many 42U racks in DC's in my time and there were a couple of rules that we never skipped: 1. Dual power in each server/device - One PSU was powered by one outlet, the other PSU by a different one with a different source meaning that we can lose a single power supply/circuit and nothing happens 2. Dual network (at minimum) - For the same reasons as above since the switches didn't always have dual power…
> I've only had a DC fail once when the engineer was performing work on the power circuitry for the DC and thought he was taking down one, but was in fact the wrong one and took both power circuits down at the same time. This is all local scale. Your setup would not survive a data center scale power outage. At scale power outages are datacenter scale. Data centers lose supply lines. They lose transformers. Sometimes…
Transfer switches at any facility that's worth being colocated in are exercised as periodically as the generators to which they connect. In all of the facilities I have had systems in (>20MW total steady state IT load), that meant once per month at minimum to keep generators happy -and to ensure the transfer functionality works-, and more often if the local grid demands it, e.g. ComEd in Chicago, or Dominion in NoVA asking for load shedding.
Re: Tell HN: AWS appears to be down again
#587Earlier quoted context omitted.
My last startup migrated from Verizon Terremark after the healthcare.gov fiasco several years ago. We also suffered from that massive outage and that was the final straw in migrating to AWS. At AWS, we built a few layers of redundant infrastructure with mulit-AZ availability within a region and then global availability across multiple regions. All this was done at roughly half the cost of the traditional hosting, eve…
Every time one of these conversations happen I end up thinking to myself that Oxide Computing needs three more competitors and a big pile of money. AWS maintains a fiction of turnkey infrastructure, and the reality of building your own is so starkly different that I haven't seen an IT group for some time that could successfully push back on these sorts of discussions. Building your own datacenter is still too much li…
I don't think putting the hardware into the rack is really the sticking point; what people like about the cloud is that it abstracts all kinds of details away for them and provides a cohesive system to manage it. AWS, Azure and Google are actually selling something like what you are talking about now [1], where for whatever legal/legacy/performance reason you need it on-prem but still want to pay AWS 5x the cost just to give you the same management interface, and they have some kind of pod they slap into your data-center.
What does it tell you that there is a market for this, where essentially what you are buying from them is a management and control plane, when other companies like BMC have been selling that as a standalone product for decades (and for the most part failing to live up to their customer's actual expectations)?
[1] https://www.bizety.com/2020/06/28/aws-outposts-google-anthos...
edit: I actually think a big pull of the cloud is also about shutting down archaic internal IT organizations that have been slowing people down so that it takes weeks and weeks to launch a simple new webservice. Better to give your programmers a cloud account and let them get shit done.
Re: Tell HN: AWS appears to be down again
#588AWS didn’t “go down”. They had an outage in one AZ, which is why there are multiple AZs in each region. If your app went down then you should be blaming your developers on this one, not AWS. Those having issues are discovering gaps in their HA designs. Obviously it’s not good for an AZ to go down but it does happen and why any production workload should be architected to have seamless failover and recover to other AZ…
> People commenting that servers shouldn’t go down ect don’t understand how true HA architectures work. You should expect and build for stuff to fail like this. Otherwise it’s like complaining that you lost data because a disk failed. Disks fail… build architecture where that won’t take you down. Is that comparison fair? If you have 2 raid-5 mirrored raid 5 boxes in your room and all disks fail at the same time, you…
More generally, any correlation between two items gives potential for a correlated failure.
Same manufacturer, same disk space, same location, same operator, same maintenance schedule, same legal jurisdiction, same planet, you name it, and there's a common failure to match
Re: Tell HN: AWS appears to be down again
#589Earlier quoted context omitted.
> Yes but if you have reliable power from two different sources then the biggest risk (I'd imagine) is the failover circuitry! Something that should be tested tbh. That's ATS. It is not really advisable to test their under load performance because the failure of an ATS would be catastrophic. ATS typically would be tested at the installation and after that their parameters would be monitored. Replacing a functional in…
> the failure of an ATS would be catastrophic Because that would mean no power at all to the DC and no way to get it back? (I am completely ignorant on this topic)
> Because that would mean no power at all to the DC and no way to get it back? (I am completely ignorant on this topic)
While most of smarts in the ATS are in the electronics, the really nasty failures come from the mechanical part.
At the end of the day a high energy ATS looks just like a switch behind a meter in your house. There's a lip that goes from one position to another, except in a high energy ATS the lip is big and when the transfer occurs it slams from one source to another.
There are only so many of those physical slams that it can withstand to being with so you want to minimize that number.
The second failure mode is that after transfer to non-main source, the lip can get stuck there, making it impossible to switch back on the main. [Once I have seem the lip melt into the secondary position. While I thought it was weird, the guys from the power company said it is not that uncommon.] This creates a massive problem as the non-main source is typically not designed for long term 24x7 operation. So now you are stuck on a secondary feeding system and you cant just transfer to main without de-energizing the system i.e. taking the power out of the entire data center.
Re: Tell HN: AWS appears to be down again
#590Earlier quoted context omitted.
Are you sure about the few minutes part? The standby generators I've seen take seconds to go from off to full load. We have an 80kw model, but I've also seen videos of load tests of much larger generators and they also take only seconds to go to full load.
It might depend on when the backup system was built. No company updates their system every year. A few minutes seems correct for one place I worked. This was back in the 90's, before UPS technology got really interesting. Our system was two large rooms with racks and racks and racks of car batteries wired together. When the power went out, the batteries took over until the diesel generator could come online. I saw it…
Lead acid batteries are still industry standard in many applications where you are OK with doing regular maintenance and you just need them to work, full stop. I think you'd be surprised how much of your power generation infrustructure, for example, has a 125VDC battery system for blackouts.