351–360 of 497 posts
Is it AWS or could it be an ISP? AWS seems to be working for me, but I’ve worked with clients in the US and spectrum internet tended to drop connections to us sporadically, which looks like an outage to our clients but is something we obviously can’t control.
Earlier quoted context omitted.
> Surely we are way past the point where someone knows how the whole thing works, all the way down. So you are disagreeing with this statement and saying that, in fact, you are the person who knows how the whole thing works? I knew tech work produced some large egos, but sheesh.
This type of person exists, and while rare, not as rare as some seem to assume.
Earlier quoted context omitted.
So long as you live in a city you are probably forgetting most of the ways to survive. I could not even try to discover which berries are edible without killing myself. However, I can teach advanced maths to a largish group of students without much trouble.
as an NYC-born, growing up with bi-monthly boy scout meetings and yearly "wilderness camps" (pitching tents in open fields, pit latrines, war games/survival, etc.) really helped fill in that gap :) i wonder if there's anything like that for adults
It's not just AWS - check the down reports: https://downdetector.com/ Cloudflare having some significant issues as well on certain domains.
HOST THE GODDAMN STATUS PAGE ON AZURE FOR FUCKS SAKE. There is zero excuse for this shit. Be professional. Acknowledge reality. It is logically impossible to run your own status page. Trying to do so just wastes everyone else on the internet's time when you have an outage.
You don’t even know what the problem is yet. Stop shouting solutions.
Earlier quoted context omitted.
So long as you live in a city you are probably forgetting most of the ways to survive. I could not even try to discover which berries are edible without killing myself. However, I can teach advanced maths to a largish group of students without much trouble.
I would recommend you buy a local book on foraging. Keep in case of emergency. But give it a read (at least the first few chapters) so you can get a basic understanding of how to forage without killing yourself. I also recommend keeping viable seeds and a camping shovel around as an insurance policy. These items aren’t in my earthquake bag (I have enough energy bars to last until the National Guard shows up). Instead…
Asking as a non-cloud-developer: why would Crunchyroll's recovery [0] lag so much behind AWS's recovery [1]? [0] https://downdetector.com/status/crunchyroll/ [1] https://downdetector.com/status/aws-amazon-web-services/
A lot of websites use a cache in front of databases (or template rendering engines, or many other systems). That cache might evict entries based on time - after 5 minutes, the entry is considered invalid.
But that means that if you have no traffic for 10 minutes, the cache completely empties. Then when traffic returns, it all skips the cache and actually triggers a real hit to the backend - which is now overwhelmed with traffic. The cache protects the backend in normal behavior, but now it's not doing its job, so the backend has many more requests than usual.
In the worst case, those requests are enqueued in a big serial sequence... but the ones at the back of the queue may time out. The client may do something like say "it's taken me 5 seconds and I still don't have a response - I'll abort and retry!" and now you have even _more_ traffic to deal with.
So cold caches and retries can conspire to keep a service down for a long time even after the root cause is fixed.
Earlier quoted context omitted.
They did add an update, faster than last time: "7:42 AM PST We are investigating Internet connectivity issues to the US-WEST-2 Region." https://status.aws.amazon.com/ Edit: They added US-WEST-1: "7:52 AM PST We are investigating Internet connectivity issues to the US-WEST-1 Region." Edit: Found root case, maybe? "8:01 AM PST We have identified the root cause of the Internet connectivity to the US-WEST-1 Region and ha…
Seems to be resolved now. And seems they hid / took away any mentioning of possible issues. Sigh.
us-west-1:
7:52 AM PST We are investigating Internet connectivity issues to the US-WEST-1 Region.
8:01 AM PST We have identified the root cause of the Internet connectivity to the US-WEST-1 Region and have taken steps to restore connectivity. We have seen some improvement to Internet connectivity in the last few minutes but continue to work towards full recovery.
8:10 AM PST We have resolved the issue affecting Internet connectivity to the US-WEST-1 Region. Connectivity within the region was not affected by this event. The issue has been resolved and the service is operating normally.
us-west-2:
7:43 AM PST We are investigating Internet connectivity issues to the US-WEST-2 Region.
8:01 AM PST We have identified the root cause of the Internet connectivity to the US-WEST-2 Region and have taken steps to restore connectivity. We have seen some improvement to Internet connectivity in the last few minutes but continue to work towards full recovery.
8:14 AM PST We have resolved the issue affecting Internet connectivity to the US-WEST-2 Region. Connectivity within the region was not affected by this event. The issue has been resolved and the service is operating normally.
I wonder if AWS will make more or less money from these outages? Will large players flee because of excessive instability? Or will smaller players go from single-AZ to more expensive multi-AZ? My guess is that no-one will leave and lots of single-AZ tenants who should be multi-AZ will use this as the impetus to do it. Honestly, having events like this is probably good for the overall resilience of distributed systems…