Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is S3 down?

news.ycombinator.com

Post reply on HN
Print view View original

651–660 of 1001 posts

Re: Ask HN: Is S3 down?

#651

Post by vocatus_gate »

rdtsc wrote: post #592
johngalt wrote: post #406

Sysadmin: I can forgive outages, but falsely reporting 'up' when you're obviously down is a heinous transgression. Somewhere a sysadmin is having to explain to a mildly technical manager that AWS services are down and affecting business critical services. That manager will be chewing out the tech because the status site shows everything is green. Dishonest metrics are worse than bad metrics for this exact reason. Any…

> but falsely reporting 'up' when you're obviously down is a heinous transgression. When SLA's are in play and so are job performance scores and bonuses there is probably a strong incentive to fudge numbers. It can be done officially ("Ah but sub-chapter 3 of chapter X in the fine print explains this wasn't technically an outage") or unofficially.

When I worked in Antarctica any outage affecting users that lasted over 50 minutes was considered an official "outage" and had to be reported to mission command. So of course ALL maintenance was rolled back/backed out if it came anywhere even close to 50 minutes, just so we wouldn't have to fill out the stupid outage paperwork.

Re: Ask HN: Is S3 down?

#652

Post by bdcravens »

I was listening to sessions from AWS Re:invent last night. What jumped out at me was the claim of 11 9's for S3. How many of those 9's have they blown through with this outage?

Re: Ask HN: Is S3 down?

#653

Post by RehnoLindeque »

jaequery wrote: post #628
carbocation wrote: post #441

Earlier quoted context omitted.

People were joking about this but it turns out to be true: they host the status icons on their service: https://twitter.com/awscloud/status/836656664635846656

Is there any service that distributes your files to multiple cloud services at the same time? With this recent S3 outage, I'm now feeling uneasy to store files on S3 for mission critical apps.

I believe that IPFS together with Filecoin is intended to be something like this in a broader, free market sense. Unfortunately IPFS is probably far from ready for mission critical apps and Filecoin hasn't launched at all.

Re: Ask HN: Is S3 down?

#654

Post by connorshea »

sotojuan wrote: post #199
connorshea wrote: post #137

Earlier quoted context omitted.

GitLab Pages [0] is pretty good for static websites IMO, you can use GitLab CI to build more complex sites as well [1]. Disclosure: I work for GitLab [0]: pages.gitlab.io [1]: https://about.gitlab.com/2016/12/07/building-a-new-gitlab-do...

GitLab really needs to provide a way to force HTTPS for domains that have it set up. I have to manually link with ` https://` everywhere.

Yeah, we've wanted that ourselves. There's an issue for it, but it's not scheduled yet. If you know Go, feel free to take a crack at it :)

Re: Ask HN: Is S3 down?

#655

Post by connorshea »

trashcan wrote: post #204
connorshea wrote: post #137

Earlier quoted context omitted.

GitLab Pages [0] is pretty good for static websites IMO, you can use GitLab CI to build more complex sites as well [1]. Disclosure: I work for GitLab [0]: pages.gitlab.io [1]: https://about.gitlab.com/2016/12/07/building-a-new-gitlab-do...

Slightly related, FYI all of the package downloads for Gitlab are timing out with a 504 error from Cloudfront ("Cloudfront is having difficulty accessing S3". The registry is also showing an error. I'm trying to downgrade to an older version because our install is not working but can't get the DEB unfortunately.

Looks like it's back up now, sorry about that :(

Re: Ask HN: Is S3 down?

#656

Post by DenisM »

z3t4 wrote: post #569
DenisM wrote: post #459

Now might be a good time to ponder a lasting solution. Clearly, we cannot trust AWS, or any other single provider, to stay up. What is the shortest, quickest to implement, path to actual high availability? You would have to host your own software which can also fail, but then at least you could do something about it. For example, you could avoid changing things during critical times of your own business (e.g. a trade…

I use a master and four mirror slaves. Then use round robin for reads. I used to have three slaves, but when two of them was down at the same time I got nervous and set up one more.

How does that work with S3? Your master and slaves are presumably just SQL databases, right? Or?

Re: Ask HN: Is S3 down?

#657

Post by Fej »

Okay, it's been a few hours and this is starting to get ridiculous. When was the last time that we had a core infrastructure outage this major, that lasted for this long?

Re: Ask HN: Is S3 down?

#658

Post by azernik »

atombender wrote: post #325

It's interesting to note the cascading effects. For example, I was immediately hit by three problems: * Slack file sharing no longer works, hangs forever (no way to hide the permanently rolling progress bar except quitting) * Github.com file uploads (e.g. dropping files into a Github issue) don't work. * Imgur.com is completely down . * Docker Hub seems to be unavailable. Can't pull/push images.

My own service was running just fine, until I tried to push out a (fairly critical) bugfix. Sadly, my deploy procedure relies on my build server pushing docker images to quay.io, and prod servers pulling them back down from quay.io, and quay.io hosts their Docker registry in S3. Time to make some apologies and excuses to my users...

Re: Ask HN: Is S3 down?

#659

Post by ta_wh »

bdcravens wrote: post #652

I was listening to sessions from AWS Re:invent last night. What jumped out at me was the claim of 11 9's for S3. How many of those 9's have they blown through with this outage?

That's a durability target, not an availability SLA. Durability != Availability.
Post reply on HN