Ask HN: Is S3 down? • Hacker News Board

Re: Ask HN: Is S3 down?

#441

Post by carbocation » Tue, Feb 28, 2017, 7:25 PM UTC

johngalt wrote: post #406

Sysadmin: I can forgive outages, but falsely reporting 'up' when you're obviously down is a heinous transgression. Somewhere a sysadmin is having to explain to a mildly technical manager that AWS services are down and affecting business critical services. That manager will be chewing out the tech because the status site shows everything is green. Dishonest metrics are worse than bad metrics for this exact reason. Any…

People were joking about this but it turns out to be true: they host the status icons on their service: https://twitter.com/awscloud/status/836656664635846656

Re: Ask HN: Is S3 down?

#442

Post by vegasje » Tue, Feb 28, 2017, 7:25 PM UTC

soccerdave wrote: post #347

vegasje wrote: post #229

We're in US-West-2 and our ELBs are dropping 5XXs like there's no tomorrow. This is definitely cascading.

We're in US-West-2 and not seeing any issues. Are the instances behind your ELB trying to access S3 in their application logic?

Nope. No S3 logic behind the scenes. A few of the ELBs are fine, and a few are not. Seems random.

The EC2 instances themselves are fine, but the affected ELBs are spitting out 500s.

Re: Ask HN: Is S3 down?

#443

Post by rebornix » Tue, Feb 28, 2017, 7:26 PM UTC

Alexa smart home component stopped working, if you try to reinstall the Alexa app on your phone, you'll find that you can't even login anymore.

Re: Ask HN: Is S3 down?

#444

Post by noir_lord » Tue, Feb 28, 2017, 7:26 PM UTC

jasoncrawford wrote: post #433

idlewords wrote: post #200

Earlier quoted context omitted.

@mikecb on Twitter explained it well. "The red icon is stored in S3 US East."

Apparently this is not a joke: “The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates.” https://twitter.com/awscloud/status/836656664635846656

I think that might be my favourite tweet this year so far.

Re: Ask HN: Is S3 down?

#445

Post by draw_down » Tue, Feb 28, 2017, 7:26 PM UTC

skiril wrote: post #364

they already admitted it: https://www.theregister.co.uk/2017/02/28/aws_is_awol_as_s3_g...

"bit-barn bods" is some of the worst alliteration I've ever seen.

Re: Ask HN: Is S3 down?

#446

Post by knaik94 » Tue, Feb 28, 2017, 7:26 PM UTC

fletom wrote: post #134

what's truly incredible is that S3 has been offline for h̶a̶l̶f̶ ̶a̶n̶ ̶h̶o̶u̶r̶ two hours now and Amazon still has the audacity to put five shiny green checkmarks next to S3 on their service page. they just now put up a box at the top saying "We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region." increased error rates? really? Amazon, everything is on fire. you are not fooling an…

So... https://twitter.com/awscloud/status/836656664635846656

"The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates."

Re: Ask HN: Is S3 down?

#447

Post by eggie5 » Tue, Feb 28, 2017, 7:26 PM UTC

all of your jokes about the dashboard not turning red b/c the icon is hosted on US EAST are true:

Amazon Web Services‏Verified account @awscloud 8m8 minutes ago More The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates.

Re: Ask HN: Is S3 down?

#448

Post by koolba » Tue, Feb 28, 2017, 7:26 PM UTC

I bet the outage is related to the new color coded CloudWatch metrics: https://twitter.com/awscloud/status/836630468778864640

As part of the release they wanted to make sure everybody gets a chance to see "red" metrics.

Re: Ask HN: Is S3 down?

#449

Post by hobofan » Tue, Feb 28, 2017, 7:26 PM UTC

wildmusings wrote: post #394

j2kun wrote: post #381

Earlier quoted context omitted.

> Amazon, everything is on fire. you are not fooling anyone Fun story, when I was an intern at Amazon there was actually a warehouse fire. The result was a lot of manual database entry updating as products were determined to be destroyed or still fit for sale.

I'm curious about what happened to products that were no longer fit for sale, but still fit for use. Do you recall?

There is "Amazon Warehouse Deals", Amazon itself acting as a used products seller on Amazon. This is usually used for returns etc., but I wouldn't be surprised if they also handle something like this.

Re: Ask HN: Is S3 down?

#450

Post by paulddraper » Tue, Feb 28, 2017, 7:27 PM UTC

carbocation wrote: post #441

johngalt wrote: post #406

Sysadmin: I can forgive outages, but falsely reporting 'up' when you're obviously down is a heinous transgression. Somewhere a sysadmin is having to explain to a mildly technical manager that AWS services are down and affecting business critical services. That manager will be chewing out the tech because the status site shows everything is green. Dishonest metrics are worse than bad metrics for this exact reason. Any…

People were joking about this but it turns out to be true: they host the status icons on their service: https://twitter.com/awscloud/status/836656664635846656

The icons aren't hosted there (or if they are, they are cached). https://status.aws.amazon.com/images/status3.gif

The status information is hosted there.