Earlier quoted context omitted.
Funny thing is, the last couple interviews I've had in Chicago and Silicon Valley, I actually get points when explaining caution is necessary when using AWS for production. A magic bullet it isn't.
Whats the alternative? Building your own is certainly not.
Amazon EC2 currently down. Affecting Heroku, Reddit, Others
261–270 of 311 posts
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#262Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#263Earlier quoted context omitted.
To be fair, AWS downtime always make the news because they affect a lot of majors websites, but that doesn't mean an average sysadmin (or devops, whatever) would do better in term of uptime with his own bay and his toys.
When our gear is down, we can actually get into the datacenter to fix it. What do you do when Amazon is down other than sweat?
But better and faster than Amazon?
I'd rather spend three hours at home saying "Shit. Well, we'll just wait for Amazon to fix that", than dropping my dinner, driving to the datacenter, and spend three hours setting up a new instance and restoring from backup.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#264Earlier quoted context omitted.
hn doesn't have a database capable of master/slave as such...so I think this will be harder if it ever becomes popular enough. I don't think it gets enough traffic it's ever likely to exceed what you can fit in a single box, from what I know.
Couldn't help but do a little digging.. In the first 99 comments of this page, average comment text size is 231 bytes. Counting all comments in articles on the front page right now, there's 1678 of them, making somewhere around 388kb of comments for the past 12 hours. So for safety's sake round that to 1mb/day and multiply by site age (5 years). That gets us 1825mb, projecting forward it's difficult to imagine a time…
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#265Unfortunate. I have to deal with a number of folks who will be overjoyed to read this news when their tech cartel vendor of choice forwards it this evening. There's a huge contingent of currently endangered infrastructure folks (and vendors who feed off them) out there who throw a party every time AWS has a visible outage.
At the going rate they'll be AA members before long.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#266Convenient that we're too backward to use AWS. That means everyone can at least talk about it here when AWS is down.
Funny thing is, the last couple interviews I've had in Chicago and Silicon Valley, I actually get points when explaining caution is necessary when using AWS for production. A magic bullet it isn't.
They recommend that people failover to other availability zones but no one puts any effort into doing it then they get annoyed when a datacenter goes offline.
Its not Amazons fault that you didn't make your service failure tolerant - its your fault!
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#267Earlier quoted context omitted.
Whats the alternative? Building your own is certainly not.
As a start, be present in multiple EC2 availability zones (not just US-east-2, basically) and regions (this is harder). Cross-region presence needn't be active-active, just a few read-only database slaves and some machines to handle SSL termination ("points of presence") for your customers on the east coast. Perform regular "fire drills" where you actually fail over live traffic and primary databases from one AZ/one…
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#268Earlier quoted context omitted.
One thing we discovered this morning: it appears the AWS console itself is hosted in N Virginia. This means that if you were trying to make changes to your EC2 instances in the West using the GUI, you couldn't, even though the instances themselves were unaffected.
shouldn't amazon themselves have architected their own app to be able to move around? I get tired of the snipes from people that "well, you're doing it wrong", as if this is trivial stuff. But if Amazon themselves aren't even making their AWS console redundant between locations, how easy/straightforward is it for anyone else? To what extent is this just "the cobbler's kids have no shoes?"
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#269Earlier quoted context omitted.
It was a joke. I made the same joke earlier today. No-one is seriously going to believe this.
I'd like to think the smiley face would make that obvious, but I guess not.
Cheers
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#270Earlier quoted context omitted.
shouldn't amazon themselves have architected their own app to be able to move around? I get tired of the snipes from people that "well, you're doing it wrong", as if this is trivial stuff. But if Amazon themselves aren't even making their AWS console redundant between locations, how easy/straightforward is it for anyone else? To what extent is this just "the cobbler's kids have no shoes?"
That's what twilio + Heroku are for, abstract up another layer. There's even a site where you just give it a github location and it does the rest.