1. Announce security vulnerability
2. People push updates as fast as possible to fix security
3. No tests, so everything blows up
Ask HN: Is S3 down?
281–290 of 1001 posts
Re: Ask HN: Is S3 down?
#282hex.pm and docker hub are both failing, a lot of projects can't CI because of these. The house of cards we built.
Re: Ask HN: Is S3 down?
#283Yup, same here. It has been a few minutes already. Wanna bet the green checkmark[1] will stay green until the incident is resolved? [1] https://status.aws.amazon.com/
I'm seeing green checkmarks across the board, but they just added a notice to the top of the page: > Increased Error Rates > We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region.
the worst thing is when your system cant handle these "increased error rates" as your control plane cascades failure due to something like this....
The worst "increased error rate" problem I had was when the API was failing and my autoscale system couldnt deal and launched thousands of instances because it couldnt tell when instances were launched (lack of API access) and the instances pummelled the fuck out of all other parts of the system and we basically had to reboot the entire platform....
Luckily, amazon is REALLY forgiving with respect to costs in these (and actually most) circumstance....
Re: Ask HN: Is S3 down?
#284Mass outage like this is exactly one of the things we are looking to avoid by building a decentralized storage grid with Sia. Sia are immune to situations like this because data is stored redundantly across dozens of servers around the world that are all running on different, unique configurations. Furthermore, there's no single central point of control on the Sia network. Sia is still under heavy development, but it…
At first I thought, "meh.." and, seriously, just the globally distributed filesystem alone is incredibly hard to make very reliable, but that's the only critical job: give data back when asked.
But then I looked at your site.. looks like bidding on surplus storage on different systems. Great idea, especially if you can ensure that people don't botnet it to death. I'm looking forward to hearing great things from you in the future.
Re: Ask HN: Is S3 down?
#285what's the SLA for s3?
And my favourite part: "To receive a Service Credit, you must submit a claim by opening a case in the AWS Support Center." -- for a company that has built itself on automation, surely you could automate some bill credits based on the SLA.
Re: Ask HN: Is S3 down?
#286Yup, same here. It has been a few minutes already. Wanna bet the green checkmark[1] will stay green until the incident is resolved? [1] https://status.aws.amazon.com/
I always joke that if one of those statuses ever went to red, it means the zombie apocalypse has begun.
In case anyone wants to see what mysterious the red icon looks like: https://status.aws.amazon.com/images/status3.gif
At best when there are problems (not like now I guess) I will see the "note" green icon https://status.aws.amazon.com/images/status1.gif
Re: Ask HN: Is S3 down?
#287Notice how Amazon.com itself is unaffected. They're a lot smarter than us.
Re: Ask HN: Is S3 down?
#288[deleted]
Re: Ask HN: Is S3 down?
#289Earlier quoted context omitted.
@mikecb on Twitter explained it well. "The red icon is stored in S3 US East."
While that may be true, that's not the reason you're seeing green. You should have been seeing a broken image or a status page not finishing loading if that was an issue.
That was (obviously) sarcasm :)
Re: Ask HN: Is S3 down?
#290what's truly incredible is that S3 has been offline for h̶a̶l̶f̶ ̶a̶n̶ ̶h̶o̶u̶r̶ two hours now and Amazon still has the audacity to put five shiny green checkmarks next to S3 on their service page. they just now put up a box at the top saying "We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region." increased error rates? really? Amazon, everything is on fire. you are not fooling an…
I don't think they intentionally kept the checkmarks there. They probably just didn't update it as quickly as developers made a post on Hacker News (not surprising, they were probably investigating).
After having seen multiple AWS outages/service disruptions, with nothing other than a green checkmark ever showing, I am now very confident that the checkmarks are hardcoded and there is no logic behind them.