Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parts of AWS are down

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 20 posts

Re: Parts of AWS are down

#11

Post by ej12n »

Bob312371 wrote: post #9
nakodari wrote: post #7

Glad to hear the issue is on Amazon's side. Our team panicked and tried to find every way to resolve this thinking it was possibly a misconfiguration at our end. I hope it's resolved soon. We're receiving complains from many users who are unable to upload their work due to this issue, I will let them know Amazon is on it.

Same here. At the time the status check on AWS was not reporting any issues and there was nothing written about it online. As a result I assumed it was an issue with my account. I only confirmed it was on their end when I noticed the favicon on the AWS console wasn't loading.

Same here, I was running around like a headless chicken and wondering what I was doing wrong when elasticbeanstalk just wouldn't deploy anything. The fact that we doubt ourselves rather than thinking aws is down is a good damn indicator how reliable they are, luckily it happened during daytime (EST anyways)

Re: Parts of AWS are down

#12

Post by nakodari »

ej12n wrote: post #11
Bob312371 wrote: post #9

Earlier quoted context omitted.

Same here. At the time the status check on AWS was not reporting any issues and there was nothing written about it online. As a result I assumed it was an issue with my account. I only confirmed it was on their end when I noticed the favicon on the AWS console wasn't loading.

Same here, I was running around like a headless chicken and wondering what I was doing wrong when elasticbeanstalk just wouldn't deploy anything. The fact that we doubt ourselves rather than thinking aws is down is a good damn indicator how reliable they are, luckily it happened during daytime (EST anyways)

The issue started happening much sooner, around morning time in EST, but it got worse as the day progressed. It looks like the Amazon engineers woke up and realized something was wrong and they updated their status page 5-6 hours late. But I agree, AWS is super reliable, it's unfortunate that this problem occurred and I hope it's resolved soon.

Re: Parts of AWS are down

#13

Post by lflux »

DDos against amazon's DNS servers. One mitigation is to use regional endpoints for bucket hosting if your website relies on serving assets - i e foo-bucket.s3.us-west-2.amazonaws.com instead foo-bucket.s3.amazonaws.com

Re: Parts of AWS are down

#14

Post by champbronc2 »

lflux wrote: post #13

DDos against amazon's DNS servers. One mitigation is to use regional endpoints for bucket hosting if your website relies on serving assets - i e foo-bucket.s3.us-west-2.amazonaws.com instead foo-bucket.s3.amazonaws.com

Where you see this is DDoS?

Re: Parts of AWS are down

#15

Post by smudgymcscmudge »

champbronc2 wrote: post #14
lflux wrote: post #13

DDos against amazon's DNS servers. One mitigation is to use regional endpoints for bucket hosting if your website relies on serving assets - i e foo-bucket.s3.us-west-2.amazonaws.com instead foo-bucket.s3.amazonaws.com

Where you see this is DDoS?

I'm not OP, but I heard that unofficially from somebody who works at aws.

Re: Parts of AWS are down

#16

Post by jared52 »

champbronc2 wrote: post #14
lflux wrote: post #13

DDos against amazon's DNS servers. One mitigation is to use regional endpoints for bucket hosting if your website relies on serving assets - i e foo-bucket.s3.us-west-2.amazonaws.com instead foo-bucket.s3.amazonaws.com

Where you see this is DDoS?

I just got off a chat with an AWS agent:

"We are investigating reports of occasional DNS resolution errors. The AWS DNS servers are currently under a DDoS attack. Our DDoS mitigations are absorbing the vast majority of this traffic, but these mitigations are also flagging some legitimate customer queries at this time. We are actively working on additional mitigations, as well as tracking down the source of the attack to shut it down. Amazon S3 customers experiencing impact from this event can update the configuration of their clients accessing S3 to specify the specific region that their bucket is in when making requests to mitigate impact. For example, instead of "mybucket.s3.amazonaws.com" a customer would instead specify "mybucket.s3.us-west-2.amazonaws.com" for their bucket in the US-WEST-2 region. If you are using the AWS SDK, you can specify the region as part of the configuration of the Amazon S3 client to make sure your requests use this region-specific endpoint name."

Re: Parts of AWS are down

#17

Post by lflux »

champbronc2 wrote: post #14
lflux wrote: post #13

DDos against amazon's DNS servers. One mitigation is to use regional endpoints for bucket hosting if your website relies on serving assets - i e foo-bucket.s3.us-west-2.amazonaws.com instead foo-bucket.s3.amazonaws.com

Where you see this is DDoS?

From the phonecall with our AWS TAM and support reps

Re: Parts of AWS are down

#19

Post by MastaX1 »

Seeing major issues here as well. All my modules were not working for the majority of the day and all images were not displaying. This is no doubt a Ddoss attack, I just don't see any other reason that the entire east coast AWS DNS servers would just stop responding all of a sudden. If the issue is not fixed by tomorrow our IT guy is gonna have a damn field day implementing all the temporary mitigation's.

Re: Parts of AWS are down

#20

Post by Bob312371 »

faCeti0us wrote: post #3

Intermittent DNS Resolution Errors We are investigating reports of occasional DNS resolution errors with Route 53 and our external DNS providers. We are actively working towards resolution. https://status.aws.amazon.com/

Why does the status page say resolved when S3 is still down for parts of US?
Post reply on HN