US-East AWS Connectivity Issues • Hacker News Board

Re: US-East AWS Connectivity Issues

#41

Post by justinsb » Fri, Sep 13, 2013, 4:00 PM UTC

tomweingarten wrote: post #20

Did anyone else notice a huge spike in incoming network traffic on their EC2 instance immediately before the outage? Roughly 9:55AM EST.

Did it look like a ddos attack, or do you think something went wrong where you were getting traffic meant for other EC2 nodes?

I'm not quite sure how you would tell the difference of course...

Re: US-East AWS Connectivity Issues

#42

Post by acdha » Fri, Sep 13, 2013, 4:01 PM UTC

jolan wrote: post #3

Amazon is continuing the trend of announcing outages 30 minutes after they start. Just signed up for a support contract since the status page said everything was fine.

“outage (n): a period when a power supply or other service is not available or when equipment is closed down”

A subset of traffic taking longer than normal is not an outage

Re: US-East AWS Connectivity Issues

#43

Post by TallboyOne » Fri, Sep 13, 2013, 4:02 PM UTC

Aaand were beck up now.

Re: US-East AWS Connectivity Issues

#44

Post by bredman » Fri, Sep 13, 2013, 4:02 PM UTC

scrabble wrote: post #4

So what is the best way to balance a hosted site between Amazon and a separate service? Because these connectivity issues suck.

One option would be to use Route 53 weighted round robin (WRR) DNS records and health checks to accomplish this.

Re: US-East AWS Connectivity Issues

#45

Post by jlgaddis » Fri, Sep 13, 2013, 4:08 PM UTC

knodi wrote: post #37

Always on a Friday...

Not just any Friday...

    $ date
    Fri Sep 13 12:07:58 EDT 2013

Re: US-East AWS Connectivity Issues

#46

Post by xdissent » Fri, Sep 13, 2013, 4:10 PM UTC

jlgaddis wrote: post #45

knodi wrote: post #37

Always on a Friday...

Not just any Friday... $ date Fri Sep 13 12:07:58 EDT 2013

Not just any Friday the 13th... http://en.wikipedia.org/wiki/Programmers'_Day

Re: US-East AWS Connectivity Issues

#47

Post by acdha » Fri, Sep 13, 2013, 4:12 PM UTC

tomweingarten wrote: post #20

Did anyone else notice a huge spike in incoming network traffic on their EC2 instance immediately before the outage? Roughly 9:55AM EST.

Nothing extraordinary but there was definitely a healthy spike in one AZ.

Re: US-East AWS Connectivity Issues

#48

Post by joe010 » Fri, Sep 13, 2013, 4:13 PM UTC

I've recently moved some of our servers over to Digital Ocean, but I'm still using AWS for DNS since their Route 53 weighted DNS with health checks work as a basic load balancer for our needs. I'm seeing DNS health checks that point at individual servers at Digital Ocean that are showing 0.91 for a status (1 being up and 0 being down. The alarms attached to the health checks keep flipping from "alarm" to "ok" and causing tons of alerts. As of about 15 minutes ago all of my checks started holding steady back at a status of 1 (ok) Good stuff :)

Re: US-East AWS Connectivity Issues

#49

Post by ihaveajob » Fri, Sep 13, 2013, 4:15 PM UTC

It looks ok now for us (appfluence.com), but even when it was down, our website was still up, only the sync services went offline. And even then, they were accessible from the web server...

Re: US-East AWS Connectivity Issues

#50

Post by TallboyOne » Fri, Sep 13, 2013, 5:05 PM UTC

xdissent wrote: post #46

jlgaddis wrote: post #45

Earlier quoted context omitted.

Not just any Friday... $ date Fri Sep 13 12:07:58 EDT 2013

Not just any Friday the 13th... http://en.wikipedia.org/wiki/Programmers'_Day

Not just any Friday the 13th Programmer's Day... http://www.holidayinsights.com/other/fortunecookie.htm