271–280 of 646 posts
I am not sure if the movement the cloud has reduced amount of failures, but it definitely has made these failures more catastrophic.
Our profession is busy makin the world less reliable and more fragile, we will have our reconning just like the shipping industry did.
Earlier quoted context omitted.
"When a fail-safe system fails, it fails by failing to fail-safe." - https://en.wikipedia.org/wiki/Systemantics
is that just playing with words?
"fail-safe" doesn't mean "doesn't fail", it means that the failure mode chooses false negatives or false positives (depending on the context) to be on the safe side.
> Due to this degradation your instance could already be unreachable
>:(
Honestly my server at home has more uptime than US-East-1
I should blog about this one day but... I have a server at OVH (not affiliated to them) which, at this point, I keep only for fun. It has 3162 days of uptime as I type this. 3 162 days. That's 8 years+ of uptime. Does it have the traffic of Amazon? No. Is it secure? Very likely not: it's running an old Debian version (Debian 7, which came out in, well, 2013). It only has one port opened though, SSH. And with quite a…
At its current use, it's likely not a major issue but imagine if someone saw this uptime and thought to take it as a statement of reliability and built a service on it. I for one, would want that disclosed because this is a disaster waiting to happen. I'd much rather someone disclose that they had a few servers each with no longer than 7 days of uptime because they'd been fully imaged and cycled in that time...
Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…
1) Can you make your on prem infrastructure go down less than Amazon's?
2) Is it worth it?
In my experience most people grossly underestimate how expensive it is to create reliable infrastructure and at the same time overestimate how important it is for their services to run uninterrupted.
--
EDIT: I am not arguing you shouldn't build your more reliable infrastructure. AWS is just a point on a spectrum of possible compromises between cost and reliability. It might not be right for you. If it is too expensive -- go for cheaper options with less reliability.
If it is too unreliable -- go build your own yourself, but make sure you are not making huge mistake because you may not understand what it actually costs to build to AWSs level.
For example, personally, not having to focus on infra reliability makes it possible for me to focus on other things that are more important to my company. Do I care about outages? Of course I do, but I understand doing this better than AWS has would cost me huge amount of focus on something that is not core goal of what we are doing. I would rather spend that time thinking how to hire/retain better people and how to make my product better.
And adding all that complexity of running this infra to my company would cause entire organisation be less flexible, which is also a cost.
So you can't look at cost of running the infra like a bill of materials for parts and services.
And if there is an outage it is good to know there is huge organisation there trying to fix it while my small organisation can focus preparing for what to do when it comes back up.
Earlier quoted context omitted.
another example of a single dc in a single AZ rendering an entire region almost unusable. This has shades of eu-central-1 all over again.
Amazon is claiming the failure is limited to a single AZ. Are you seeing failures for instances outside of that AZ? If not, how has this rendered "the entire region almost unusable"?
Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…
I must admit that I do always try and maintain a separate data backup for true disaster recovery scenarios - but those are mainly focused around AWS locking me out of our AWS account (and hence we can't access our data or backups) or recovering from a crypto scam hack that also corrupts on-platform backups, for example.
Question to the sysadmins here: Is it really that outrageous of amazon to have such issues or are people way to spoiled to appreciate the effort that goes into maintaining such a service? Edit: Not supporting amazon, i generally dislike the company. I just don't understand the extend to which the criticism is justified
1. Did AMZN build an appropriate architecture?
2. Did AMZN properly represent that architecture in both documentation and sales efforts?
3. What the heck is going on with AMZN?
Let's say that they build an environment in which power is not fully redundant and tested at the rack level, but is fully redundant and tested across multiple availability zones. Did they then issue statements of reliability to their prospective and existing customers saying that a single availability zone does not have redundant power, and customers must duplicate functionality in at least 2 AZs to survive a SPOF?
If you haven't seen yet, news is it was a power loss: > 5:01 AM PST We can confirm a loss of power within a single data center within a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. This is affecting availability and connectivity to EC2 instances that are part of the affected data center within the affected Availability Zone. We are also experiencing elevated RunInstance API error rates for launches wi…
How come they dont have power backups?
I once had to prepare for a total blackout scenario in a datacenter because there was a fault in the power supply system that required bypassing major systems to fix. Had some mistake or fault happened during those critical moments, all power would've been lost.
Well-designed redundancy makes high-impact incidents less likely, but you're not immune to Murphy's law.
Earlier quoted context omitted.
another example of a single dc in a single AZ rendering an entire region almost unusable. This has shades of eu-central-1 all over again.
Amazon is claiming the failure is limited to a single AZ. Are you seeing failures for instances outside of that AZ? If not, how has this rendered "the entire region almost unusable"?