Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Keep a static "emergency mode" site on S3

coderwall.com

Post reply on HN
Print view View original

61–70 of 85 posts

Re: Keep a static "emergency mode" site on S3

#61

Post by colmmacc »

jedberg wrote: post #59
colmmacc wrote: post #29

(I work on Route 53). As __lucas has mentioned, this can be achieved with Route 53 Failover, which we'd recommend. Route 53 Failover is (hopefully) pretty easy to configure. Just mark your ELB as the primary and enable target healthchecks, and add the S3 website bucket as the secondary. We'd also suggest that you use an S3 website bucket hosted in a different region than your ELB. This should take no more than a minu…

Except that his way all the traffic instantly switches, and your way you have to wait for DNS propagation, which about 15% of the users on the internet will not pick up for over a week. DNS is an awful way to do failover.

To your point ; it's ok to do both.

Route 53 supports DNS TTLs as low as 0 seconds. ELB and S3 endpoints both have 60 second TTLs. My experience with flipping names like www.amazon.com doesn't reflect the 15% figure. I've seen about 97% of web traffic honouring the TTL and flipping quickly. Within 5 minutes almost all of the rest too. We also take CloudFront sites in and out of service for maintenance, and in 5 years I've never anything like a 15% straggler effect.

That said, we do see a very small number of stragglers. While resolvers over-riding TTLs hasn't shown up as a significant problem, buggy clients can be; we come across clients now and then who either don't re-resolve ever (Various JVMs and their infinite caches are a common cause), or only re-resolve on failures (which is fine for failover, but not great for traffic management).

If you have a distribution time plot for the 15% figure it'd be interesting to see; https://www.dns-oarc.net/ would be a good venue, https://lists.dns-oarc.net/mailman/listinfo/dns-operations is the open list. Ignoring TTLs for a week is very concerning; it would very likely break many DNSSEC configurations. Is it possible you were dealing with robots?

Re: Keep a static "emergency mode" site on S3

#62

Post by EGreg »

Retric wrote: post #51
EGreg wrote: post #35

Earlier quoted context omitted.

Why don't you just always have your site static and connect to your back end as necessary? Treat it as a webservice with uptime, etc.

Because designing V0.1 of your application from the ground up based on edge cases is a Great way to never release anything. Spending a weekend setting up a static failover on the other hand has no long term downside and let's you put off worrining about a host of those edege caeses without any real down side. It's like buying a UPS for your dev box it's probably never going to matter, but it's cheap so feel free.

Actually it's a great way to simultaneously design a website AND an api for others to use. It's also a great way to separate concerns. It's also a great way to reduce load on your server. In fact, it's an easy way to have some people code a standard back end with a standard authentication so that some other people can make front ends for the web, iphone, and more. You can use, for example, oauth to authenticate with the back end, from any front end app.

http://www.discourse.org/ is one example of such an approach

Re: Keep a static "emergency mode" site on S3

#65

Post by Bockit »

adrr wrote: post #48
mayop100 wrote: post #34

Or better yet -- make your entire site static to begin with! This is how our site works (firebase.com). Our entire site is static content that's generated at deploy time and hosted on a CDN. Dynamic data is loaded asynchronously as needed. If a server were to go down, at least all of the static pieces (which is most of the site) would be unaffected. We use Firebase to power the dynamic portions (obviously), but you c…

You still need dynamic "pages". Things like geoip redirection or localization etc. You could in theory load this stuff via ajax, but this doesn't work for search engine crawlers. But i do agree most stuff can be static, you can push json on fragments onto s3 and have the web page fetch them via ajax.

We've done this for a lot of data vis work. Clients have access to a cms which lets them stage and publish their data. Doing so puts JSON files on s3 where we also serve the site. There are some trade offs, sometimes you miss having that rest api, but you also gain a lot too.

Re: Keep a static "emergency mode" site on S3

#66

Post by dangoor »

EGreg wrote: post #7

Yeah but the big question is, how can you switch the DNS over in time when NONE of your servers can respond fast enough?

Didn't the original article mention setting Apache to redirect the requests to the static site?

I would imagine that Apache (running on something better than a C64) can handle an absolutely ridiculous amount of traffic that it's just redirecting.

Re: Keep a static "emergency mode" site on S3

#67

Post by ra »

colmmacc wrote: post #29

(I work on Route 53). As __lucas has mentioned, this can be achieved with Route 53 Failover, which we'd recommend. Route 53 Failover is (hopefully) pretty easy to configure. Just mark your ELB as the primary and enable target healthchecks, and add the S3 website bucket as the secondary. We'd also suggest that you use an S3 website bucket hosted in a different region than your ELB. This should take no more than a minu…

Thanks this is a really great idea. I'm going to try this out.

Re: Keep a static "emergency mode" site on S3

#68

Post by VexXtreme »

colmmacc wrote: post #29

(I work on Route 53). As __lucas has mentioned, this can be achieved with Route 53 Failover, which we'd recommend. Route 53 Failover is (hopefully) pretty easy to configure. Just mark your ELB as the primary and enable target healthchecks, and add the S3 website bucket as the secondary. We'd also suggest that you use an S3 website bucket hosted in a different region than your ELB. This should take no more than a minu…

We use weighted A records for this purpose. We basically have two weighted A records, one pointing to the load balancer and the other one to the static maintenance site on S3. When we want to switch sites, we just flip their weights (they are always 1 and 0), hence enforcing 100% redirection. It's not a perfect system but it works and DNS propagation is usually done within 15 minutes (haven't tried modifying the TTLs). Haven't been able to come up with anything simpler than this.

Re: Keep a static "emergency mode" site on S3

#69

Post by jafaku »

twistedpair wrote: post #56
chc wrote: post #46

Earlier quoted context omitted.

One man says, "Make your whole site static! Load content with AJAX instead of generating HTML on the server!" The other man says, "Your site doesn't work on my browser with NoScript." These two men will never like each other.

The NoScript man is used to this and his fitted tinfoil hat. If he's more lenient he uses Ghostery to block all 3rd party addons and various cookie blockers to keep out the cruft, but keeps the actual site unperturbed.

True tinfoils don't use propietary software.

Re: Keep a static "emergency mode" site on S3

#70

Post by btilly »

colmmacc wrote: post #61
jedberg wrote: post #59

Earlier quoted context omitted.

Except that his way all the traffic instantly switches, and your way you have to wait for DNS propagation, which about 15% of the users on the internet will not pick up for over a week. DNS is an awful way to do failover.

To your point ; it's ok to do both. Route 53 supports DNS TTLs as low as 0 seconds. ELB and S3 endpoints both have 60 second TTLs. My experience with flipping names like www.amazon.com doesn't reflect the 15% figure. I've seen about 97% of web traffic honouring the TTL and flipping quickly. Within 5 minutes almost all of the rest too. We also take CloudFront sites in and out of service for maintenance, and in 5 years…

I ran across the "DNS broken for a week", and it was broken for ISPs that run their own DNS for home customers. They didn't want to keep asking for upstream DNS, so just ignored the ttl.

But I ran across this around the year 2000. And the biggest offender was named AOL. I could well believe that it is very, very different today.

Post reply on HN