No post body was provided.
Amazon EC2 currently down. Affecting Heroku, Reddit, Others
251–260 of 311 posts
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#252Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#253The tech industry is manufacturing its version of "Too big to fail" entity.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#254Earlier quoted context omitted.
Funny thing is, the last couple interviews I've had in Chicago and Silicon Valley, I actually get points when explaining caution is necessary when using AWS for production. A magic bullet it isn't.
Whats the alternative? Building your own is certainly not.
As a start, be present in multiple EC2 availability zones (not just US-east-2, basically) and regions (this is harder). Cross-region presence needn't be active-active, just a few read-only database slaves and some machines to handle SSL termination ("points of presence") for your customers on the east coast. Perform regular "fire drills" where you actually fail over live traffic and primary databases from one AZ/one region to another.
"Building your own" is also something very few people (including Amazon itself up until fairly late, probably after the IPO) do: you can use a managed hosting provider (very common, usually cheaper than EC2) or lease colo space (which doesn't imply maintaining on-site personnel in the leased space: most colos provide "remote hands"). You can still use EC2 for async processing and offline computation, S3 for blob storage, etc... or even S3 for "points of presence" on different US coasts, Asia/Pacific, Europe, but run databases, et al in a leased colo or a managed hosting provider.
Yes, these options are more expensive than running a few instances in a single EC2 AZ: but that's the price of offering high availability SLA to your customers. It's a business decision.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#255Earlier quoted context omitted.
Funny thing is, the last couple interviews I've had in Chicago and Silicon Valley, I actually get points when explaining caution is necessary when using AWS for production. A magic bullet it isn't.
Whats the alternative? Building your own is certainly not.
As a start, be present in multiple EC2 availability zones (not just US-east-2, basically) and regions (this is harder). Cross-region presence needn't be active-active, just a few read-only database slaves and some machines to handle SSL termination ("points of presence") for your customers on the east coast. Perform regular "fire drills" where you actually fail over live traffic and primary databases from one AZ/one region to another.
"Building your own" is also something very few people (including Amazon itself up until fairly late, probably after the IPO) do: you can use a managed hosting provider (very common, usually cheaper than EC2) or lease colo space (which doesn't imply maintaining on-site personnel in the leased space: most colos provide "remote hands"). You can still use EC2 for async processing and offline computation, S3 for blob storage, etc... or even S3 for "points of presence" on different US coasts, Asia/Pacific, Europe, but run databases, et al in a leased colo or a managed hosting provider.
Yes, these options are more expensive than running a few instances in a single EC2 AZ: but that's the price of offering high availability SLA to your customers. It's a business decision.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#256Earlier quoted context omitted.
Funny thing is, the last couple interviews I've had in Chicago and Silicon Valley, I actually get points when explaining caution is necessary when using AWS for production. A magic bullet it isn't.
Whats the alternative? Building your own is certainly not.
As a start, be present in multiple EC2 availability zones (not just US-east-2, basically) and regions (this is harder). Cross-region presence needn't be active-active, just read-only database slaves and perhaps some machines to handle SSL termination ("points of presence") for your customers on the west coast. Perform regular "fire drills" where you actually fail over live traffic and primary databases from one AZ/one region to another.
This is what Quora, Netflix, and few others doe now I believe. Note, how whenever there's an EC2 outage there are almost always a few customers which are not affected.
"Building your own" is also something very few people (including Amazon itself up until fairly late, probably after the IPO) do: you can use a managed hosting provider (very common, usually cheaper than EC2) or lease colo space (which doesn't imply maintaining on-site personnel in the leased space: most colos provide "remote hands"). You can still use EC2 or EMR for async processing and offline computation, S3 for blob storage, etc... or even S3 for "points of presence" on different US coasts, Asia/Pacific, Europe, but run databases, et al in a leased colo or a managed hosting provider.
Yes, these options are more expensive than running a few instances in a single EC2 AZ: but that's the price of offering high availability SLA to your customers. It's a business decision: the benefit of "the cloud" is to enable you to quickly reach a point where you can afford to make such decisions rather than pay hosting costs up-front and risk running out of money before you fully understand what kind of infrastructure and SLAs you need.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#257Building infrastructure on top of Amazon with global replication across multiple availability zones can sidestep such failures and guarantee uninterrupted operations to customers - www.pubnub.com is unaffected by AWS being down.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#258Earlier quoted context omitted.
Funny thing is, the last couple interviews I've had in Chicago and Silicon Valley, I actually get points when explaining caution is necessary when using AWS for production. A magic bullet it isn't.
Whats the alternative? Building your own is certainly not.
Old school colo/dedicated servers/etc. There's something delightfully simple about only having to deal with "standard" hardware failures.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#259Earlier quoted context omitted.
Until it can't. Then five years later you get Spanner.
You can buy some seriously big boxes, and easily split off a lot of services onto multiple boxes. The big problem with the "single big box" strategy is being able to do upgrades -- I see hn go down frequently for 5-10 min at a time in the middle of the night, which I assume is upgrades/reboots. The happy medium is probably splitting database (master/slave at least) and cdn (if needed) and some other services (AAA? lo…
The Arc process hosting HN blows up at least once an hour (I wouldn't be surprised if there was a cronjob restarting it) and much more frequently in peak usage periods.
You wouldn't notice if it weren't for the use of closures for every form and all pagination, every time the process dies all of them are invalid (except in the rare case that they lead to a random new place!).
There's no database, everything is in-memory loaded on-demand from flat files. That wouldn't be so bad except that it's all then addressed by the memory locations rather than the content identifiers! There can be only one server per app, and to keep it real interesting PG hosts all the apps on the same box, during YC application periods he regularly limits HN to keep the other apps more available.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#260Convenient that we're too backward to use AWS. That means everyone can at least talk about it here when AWS is down.
I'm not familiar with HN's technical stack (other than arc), how has it scaled as the community grew over the years?
Well, for one; the site is basically an engine for rendering out ..., without much in the way of complex and frequent client-side requests.
(this isn't to downplay the challenges faced by scaling a site with the amount of traffic HN gets)