Keep a static "emergency mode" site on S3 • Hacker News Board

Re: Keep a static "emergency mode" site on S3

#31

Post by brianmcconnell » Fri, Jul 12, 2013, 5:34 PM UTC

App Engine is also a good way to do this. Let's say you have the bulk of your site running on AWS (assume for whatever reason you don't want to use GAE as your primary environment)

* Have a heartbeat task on GAE that polls your server, and if its running, write to "serverisrunning" on memcache with a short time to live.

* (normal operation) : redirect initial visitors to your AWS site.

* (server not responding to a heartbeat task on GAE, or memcache miss), serve static content from GAE, or a limited functionality version of your app hosting on GAE (for example, a static site with signup form).

This type of setup has the added bonus of automatically detecting an outage and responding to it. While App Engine has its own downtime issues, outages are transient. Since they migrated to the high replication data store, I haven't seen anything that lasted more than a few minutes.

Re: Keep a static "emergency mode" site on S3

#32

Post by edwintorok » Fri, Jul 12, 2013, 5:34 PM UTC

cenhyperion wrote: post #17

kmfrk wrote: post #5

As someone said in the comments, just keep in mind that S3 is just for storage , not serving. You'll need something like CloudFront for that, although I don't know at which degree of activity it's going to save you money to use it. Maybe from the get-go?

That's completely false. Hosting a static website on S3 is very well documented.

And how reliable is it?

There have been reports about S3 returning 50x errors, when (as documented) the client should retry. A browser is not an S3 client, so it won't retry ... https://news.ycombinator.com/item?id=4976893 https://news.ycombinator.com/item?id=4977360 https://news.ycombinator.com/item?id=4981482 https://news.ycombinator.com/item?id=2897959

Re: Keep a static "emergency mode" site on S3

#33

Post by RyanGWU82 » Fri, Jul 12, 2013, 5:36 PM UTC

diminoten wrote: post #25

> As far as I'm concerned, S3 static file website serving is completely indestructible. I only need one bland Apache server to bump requests over to it. Is this generally the experience of everyone here?

S3's not completely indestructible, but it is the most reliable platform I've encountered. During the ~1.5 years I was at Pinterest, we used S3 to store and serve all of the cached images on our site, with CDNs in front of course. We only ran into 2 incidents with S3, and only one of those would have affected "normal" customers. (The other was our own fault -- we suddenly started writing hundreds of files per second to a bucket that wasn't ready for this level of traffic.)

Re: Keep a static "emergency mode" site on S3

#34

Post by mayop100 » Fri, Jul 12, 2013, 5:40 PM UTC

Or better yet -- make your entire site static to begin with! This is how our site works (firebase.com). Our entire site is static content that's generated at deploy time and hosted on a CDN. Dynamic data is loaded asynchronously as needed. If a server were to go down, at least all of the static pieces (which is most of the site) would be unaffected.

We use Firebase to power the dynamic portions (obviously), but you can use plain old AJAX requests as well.

The age of the dynamically-generated HTML page is coming to an end.

Re: Keep a static "emergency mode" site on S3

#35

Post by EGreg » Fri, Jul 12, 2013, 5:44 PM UTC

Retric wrote: post #16

EGreg wrote: post #7

Yeah but the big question is, how can you switch the DNS over in time when NONE of your servers can respond fast enough?

You still need to update DNS ASAP, but unless your dealing with a physical flood/fire you can often get something to respond on the old IP. So, depending on the type of failure your dealing with, often setting up a static redirect is viable for insane levels of traffic even if your hosting it off of a single underpowered CPU and limited bandwidth. Aka the site has been running off your FIOS connection and a spare CPU…

Why don't you just always have your site static and connect to your back end as necessary? Treat it as a webservice with uptime, etc.

Re: Keep a static "emergency mode" site on S3

#36

Post by mnutt » Fri, Jul 12, 2013, 5:55 PM UTC

colmmacc wrote: post #29

(I work on Route 53). As __lucas has mentioned, this can be achieved with Route 53 Failover, which we'd recommend. Route 53 Failover is (hopefully) pretty easy to configure. Just mark your ELB as the primary and enable target healthchecks, and add the S3 website bucket as the secondary. We'd also suggest that you use an S3 website bucket hosted in a different region than your ELB. This should take no more than a minu…

Do you guys have any plans to actually show the results of the DNS changes based on health checks? Connecting the health checks to cloudwatch is a great start, but it would be really useful to actually see which of my rules are in effect at any given time.

Re: Keep a static "emergency mode" site on S3

#37

Post by Pxtl » Fri, Jul 12, 2013, 5:56 PM UTC

I'm surprised this forum isn't full of the usual "omg, use nginx instead of apache for rerouting".

Re: Keep a static "emergency mode" site on S3

#38

Post by krallin » Fri, Jul 12, 2013, 6:06 PM UTC

mayop100 wrote: post #34

Or better yet -- make your entire site static to begin with! This is how our site works (firebase.com). Our entire site is static content that's generated at deploy time and hosted on a CDN. Dynamic data is loaded asynchronously as needed. If a server were to go down, at least all of the static pieces (which is most of the site) would be unaffected. We use Firebase to power the dynamic portions (obviously), but you c…

Here's a small, shameless, open source, plug: I work on the "Cactus" static website generator, check out and fork our work here: github.com/koenbok/Cactus (current version), https://github.com/krallin/Cactus (next version)

Re: Keep a static "emergency mode" site on S3

#39

Post by reeses » Fri, Jul 12, 2013, 6:06 PM UTC

gregd wrote: post #12

Why not just do an origin-pull via CloudFront? No need to build a static site on S3.

You would need to make sure you still build the 'static site generator' on your current site (so login, search, and any other functionality dependent on your app is not exposed).

This is relatively easy, and could possibly even just use CSS with the understanding that yes, someone could have a bad experience.

Re: Keep a static "emergency mode" site on S3

#40

Post by troels » Fri, Jul 12, 2013, 6:08 PM UTC

mayop100 wrote: post #34

Or better yet -- make your entire site static to begin with! This is how our site works (firebase.com). Our entire site is static content that's generated at deploy time and hosted on a CDN. Dynamic data is loaded asynchronously as needed. If a server were to go down, at least all of the static pieces (which is most of the site) would be unaffected. We use Firebase to power the dynamic portions (obviously), but you c…

Agree. Another benefit of having the site mainly static, with dynamic parts loaded over ajax is that it becomes much easier to apply efficient http-level caching.