Live data from Hacker News

Amazon EC2 currently down. Affecting Heroku, Reddit, Others

status.aws.amazon.com

261–270 of 311 posts

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#261

Earlier quoted context omitted.

Funny thing is, the last couple interviews I've had in Chicago and Silicon Valley, I actually get points when explaining caution is necessary when using AWS for production. A magic bullet it isn't.

Whats the alternative? Building your own is certainly not.

Building your own what?

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#263

Earlier quoted context omitted.

To be fair, AWS downtime always make the news because they affect a lot of majors websites, but that doesn't mean an average sysadmin (or devops, whatever) would do better in term of uptime with his own bay and his toys.

When our gear is down, we can actually get into the datacenter to fix it. What do you do when Amazon is down other than sweat?

> we can actually get into the datacenter to fix it.

But better and faster than Amazon?

I'd rather spend three hours at home saying "Shit. Well, we'll just wait for Amazon to fix that", than dropping my dinner, driving to the datacenter, and spend three hours setting up a new instance and restoring from backup.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#264
post #224

Earlier quoted context omitted.

hn doesn't have a database capable of master/slave as such...so I think this will be harder if it ever becomes popular enough. I don't think it gets enough traffic it's ever likely to exceed what you can fit in a single box, from what I know.

Couldn't help but do a little digging.. In the first 99 comments of this page, average comment text size is 231 bytes. Counting all comments in articles on the front page right now, there's 1678 of them, making somewhere around 388kb of comments for the past 12 hours. So for safety's sake round that to 1mb/day and multiply by site age (5 years). That gets us 1825mb, projecting forward it's difficult to imagine a time…

Clearly pg should release the dataset and institute a annual round of hn golf where participants compete by recreating hn and trying to get the best performance for a given (changing) deployment target (SSD vs HDD, different RAM & CPU).

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#265

Unfortunate. I have to deal with a number of folks who will be overjoyed to read this news when their tech cartel vendor of choice forwards it this evening. There's a huge contingent of currently endangered infrastructure folks (and vendors who feed off them) out there who throw a party every time AWS has a visible outage.

> who throw a party every time AWS has a visible outage.

At the going rate they'll be AA members before long.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#266
post #192

Convenient that we're too backward to use AWS. That means everyone can at least talk about it here when AWS is down.

Funny thing is, the last couple interviews I've had in Chicago and Silicon Valley, I actually get points when explaining caution is necessary when using AWS for production. A magic bullet it isn't.

I can't feel that Amazon are a bit of a Cassandra (mythological not the software) when these outages occur.

They recommend that people failover to other availability zones but no one puts any effort into doing it then they get annoyed when a datacenter goes offline.

Its not Amazons fault that you didn't make your service failure tolerant - its your fault!

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#267
post #254

Earlier quoted context omitted.

Whats the alternative? Building your own is certainly not.

As a start, be present in multiple EC2 availability zones (not just US-east-2, basically) and regions (this is harder). Cross-region presence needn't be active-active, just a few read-only database slaves and some machines to handle SSL termination ("points of presence") for your customers on the east coast. Perform regular "fire drills" where you actually fail over live traffic and primary databases from one AZ/one…

We run gear in multiple physical locations, but both the application and data is stored/backed in S3. This allows us the redundancy of S3 without the cost and fragile nature of EC2.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#268

Earlier quoted context omitted.

One thing we discovered this morning: it appears the AWS console itself is hosted in N Virginia. This means that if you were trying to make changes to your EC2 instances in the West using the GUI, you couldn't, even though the instances themselves were unaffected.

shouldn't amazon themselves have architected their own app to be able to move around? I get tired of the snipes from people that "well, you're doing it wrong", as if this is trivial stuff. But if Amazon themselves aren't even making their AWS console redundant between locations, how easy/straightforward is it for anyone else? To what extent is this just "the cobbler's kids have no shoes?"

That's what twilio + Heroku are for, abstract up another layer. There's even a site where you just give it a github location and it does the rest.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#269

Earlier quoted context omitted.

It was a joke. I made the same joke earlier today. No-one is seriously going to believe this.

I'd like to think the smiley face would make that obvious, but I guess not.

When I commented the two other comments replying to yours were taking it seriously, so I decided to take it seriously too. Text is a pretty bad medium for hearing tone, sorry I misinterpreted yours; for some reason the "the cynical side of me" bit made me think your comment wasn't entirely in jest.

Cheers

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#270

Earlier quoted context omitted.

shouldn't amazon themselves have architected their own app to be able to move around? I get tired of the snipes from people that "well, you're doing it wrong", as if this is trivial stuff. But if Amazon themselves aren't even making their AWS console redundant between locations, how easy/straightforward is it for anyone else? To what extent is this just "the cobbler's kids have no shoes?"

That's what twilio + Heroku are for, abstract up another layer. There's even a site where you just give it a github location and it does the rest.

Well the Heroku abstraction was leaking like a sieve today.
Post reply on HN