Reminder that the internet was literally invented to avoid this kind of nuclear attack. But i guess people are herdish animals and prefer to die as a group
Seems like the Internet level networking is quite robust at this point.
Tell HN: AWS appears to be down again
441–450 of 497 posts
Re: Tell HN: AWS appears to be down again
#442Earlier quoted context omitted.
They did add an update, faster than last time: "7:42 AM PST We are investigating Internet connectivity issues to the US-WEST-2 Region." https://status.aws.amazon.com/ Edit: They added US-WEST-1: "7:52 AM PST We are investigating Internet connectivity issues to the US-WEST-1 Region." Edit: Found root case, maybe? "8:01 AM PST We have identified the root cause of the Internet connectivity to the US-WEST-1 Region and ha…
someone tripped over the fiber run i bet. Or, a cleaning person unplugged a router to plugin a vacuum (that actually happened but to a minicomputer iirc)
No way a cleaning person can do that in a datacenter.
Re: Tell HN: AWS appears to be down again
#443Earlier quoted context omitted.
Curios if you tell your customers you’re totally ok with having lower than 99.9 availability
How many 9s can you get from a single-region multi-AZ deployment not on us-east-1 and which nly uses basic services (EC2, IAM, S3, DynamoDB, etc)? Really only 3?
Depends on how critical they are to your stack. Ime if you use more than a few products and either one of them can take you down yeah it’s less than 3. Just something to ponder but if s3 didn’t meet 99.9 for the month you get a whopping 10% back. Other cloud vendors aren’t much better at this (actually worse). Not even to mention that you need to leave some room for your own fuckups
Re: Tell HN: AWS appears to be down again
#444Earlier quoted context omitted.
Taking badges out of the cloud reduces points of failure by several orders of magnitude. Cloud-based badges make sense if you have locations with small staffs and no HR people or managers. Like if you're controlling access to a microwave tower on the top of a mountain. But badges-in-the-cloud for an office building full of people who are being supervised by supposedly trusted managers, and all of whom has been vetted…
> Taking badges out of the cloud reduces points of failure by several orders of magnitude. I'm not convinced that's true, or at least certainly not an order of magnitude. Wouldn't a badge system hosted on-prem also need a user management system (database), a hosted management interface, have a dependency on the LAN, and need most of the same hardware? Such a system would also need to be running on a local server(s),…
All of those things would also be needed by the cloud provider, too. Just because it's on-prem doesn't mean it doesn't need servers, power conditioning, physical security, etc. "Cloud" isn't magic fairies. It's just renting someone else's points of failure.
In addition, you're forgetting the thousands of points of failure between the building and the cloud provider. Everything from routers being DDOSed by script kiddies to ransomware gangs attacking infrastructure to Phil McCracken slicing a fiber line with his new post hole digger.
Re: Tell HN: AWS appears to be down again
#445I wonder if AWS will make more or less money from these outages? Will large players flee because of excessive instability? Or will smaller players go from single-AZ to more expensive multi-AZ? My guess is that no-one will leave and lots of single-AZ tenants who should be multi-AZ will use this as the impetus to do it. Honestly, having events like this is probably good for the overall resilience of distributed systems…
* Free chaos monkey installed in every AZ
> * Free chaos monkey installed in every AZ
Only during this beta period, AWS will start charging for this feature soon enough.
Re: Tell HN: AWS appears to be down again
#446Earlier quoted context omitted.
AWS wouldn't monitor itself from a competitor, of course Why not? The big tech companies use each other all the time. For example, set up a new firewall on macOS and you can see how many times Apple pulls data from Amazon or Azure or other competitors' APIs and services.
Apple is not a competitor to AWS or Azure in any way. They offer not infrastructure/platform as a service that I am aware of.
Apple and Amazon are competitors. Apple and Microsoft competitors.
The postulation was that Apple and Amazon weren't competitors. Not that they're not competitors in a specific niche.
Re: Tell HN: AWS appears to be down again
#447Earlier quoted context omitted.
... because you need to contact a server 1000 miles away to issue badges in your building. This cloud-for-everything-even-local-devices thing is both hilarious and sad. I wonder if anyone had trouble doing their dishes or laundry today, because I'm sure someone thought dish washers and washing machines needed cloud.
Yes, everyone but you is wrong. Many logical people have decided to abstract away their soul-crushing anxieties and legal gray area during outages to incredibly stable and well-staffed cloud infrastructure providers. If you and your team are better at taking care of hardware than an entire building full of highly paid engineering specialists, then that's cool for you, but also, no you're not. That's not to say you're…
There is absolutely no reason for a local device (like a door lock or dishwasher as per OP) to depend on any external connectivity. Not to the company on-prem hardware, not to AWS.
Re: Tell HN: AWS appears to be down again
#448Earlier quoted context omitted.
well, atleast you have the agency to do something about it yourself. also, building access systems should be hosted in the building they reside in for security reasons anyways.
This creates some really fun failure cases on the form of "I need to enter the building so anybody can enter the building". Depending on the cloud is certainly a very stupid decision. keeping everything inside the building is better, but still not ideal.
Any electronic access system like this requires manual backup. As in, some doors with regular locks using physical keys.
Re: Tell HN: AWS appears to be down again
#449I wonder if AWS will make more or less money from these outages? Will large players flee because of excessive instability? Or will smaller players go from single-AZ to more expensive multi-AZ? My guess is that no-one will leave and lots of single-AZ tenants who should be multi-AZ will use this as the impetus to do it. Honestly, having events like this is probably good for the overall resilience of distributed systems…
If my company is any indication, they're going to make more money since everyone will simply check the multi-AZ or multi-region checkboxes they didn't before and throw more money at the problem instead of doing proper resiliency engineering themselves.
Re: Tell HN: AWS appears to be down again
#450Earlier quoted context omitted.
Taking badges out of the cloud reduces points of failure by several orders of magnitude. Cloud-based badges make sense if you have locations with small staffs and no HR people or managers. Like if you're controlling access to a microwave tower on the top of a mountain. But badges-in-the-cloud for an office building full of people who are being supervised by supposedly trusted managers, and all of whom has been vetted…
> Taking badges out of the cloud reduces points of failure by several orders of magnitude. I'm not convinced that's true, or at least certainly not an order of magnitude. Wouldn't a badge system hosted on-prem also need a user management system (database), a hosted management interface, have a dependency on the LAN, and need most of the same hardware? Such a system would also need to be running on a local server(s),…
The remote solution requires all of those same things, plus in addition it requires internet connectivity to be up and reliable, the cloud provider be available and the third party company be up and still in business.
Adding complexity and moving parts never reduces points of failure. It can reduce daily operating worries as long as everything works, but it can't reduce points of failure. It also means than someday when it breaks, the root causes will be more opaque.