Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

471–480 of 497 posts

Re: Tell HN: AWS appears to be down again

#471

Post by dogecoinbase »

gizmodo59 wrote: post #462
kenhwang wrote: post #449

Earlier quoted context omitted.

If my company is any indication, they're going to make more money since everyone will simply check the multi-AZ or multi-region checkboxes they didn't before and throw more money at the problem instead of doing proper resiliency engineering themselves.

It doesn’t matter how much of resiliency engineering you do. Having everything in a single AZ is a risk. If this is acceptable then it’s fine if not you need to think of multi az from day 1.

Auth0 ran in six AZs in two regions[1] and went down today[2], because they picked the wrong two regions. How many regions and AZs should someone pay for before they get reliability?

1: https://auth0.com/blog/auth0-architecture-running-in-multipl... 2: https://twitter.com/auth0/status/1471159935597793290

Re: Tell HN: AWS appears to be down again

#472

Post by spenczar5 »

CoastalCoder wrote: post #386
spenczar5 wrote: post #358

Earlier quoted context omitted.

I don't know for sure, but this is generally common because caches get cold. A lot of websites use a cache in front of databases (or template rendering engines, or many other systems). That cache might evict entries based on time - after 5 minutes, the entry is considered invalid. But that means that if you have no traffic for 10 minutes, the cache completely empties. Then when traffic returns, it all skips the cache…

I'm accustomed with cache-eviction policies based on LRU, age, etc. But in my systems, eviction happens only when (a) the content is known to be invalid, or (b) there's competition for cache space. IIUC the parent comment, it's describing a policy that evicts entries even (a) and (b) are false. Is that common in the web-hosting / CDN world? Or is age considered a proxy for stale?

Right, age is used as a proxy for stale, because we often don't have anything better.

A lot of web systems work this way - DNS records for example use a "TTL" which means "time to live." If the TTL is 60, then you throw it out of the cache after 60 seconds even if you have room in the cache, and you have no reason to believe it's invalid. This lets independent entities (like a DNS authority) make a change and get it rolled out everywhere.

I think the reason this is common is that proving cache invalidity is so hard, especially with the typical "dumb" cache appliances that are widely used. They just do stuff like cache the response bytes for a particular URL; they might not even understand HTTP beyond interpreting the request's headers, and certainly don't really understand the response.

Re: Tell HN: AWS appears to be down again

#473

Post by electroly »

account758 wrote: post #204

AWS Global Accelerator not working correctly anymore as well, connections dropped worldwide. Seems like it is managed from us-west-2 and not redundant.

This comment taught me about the existence of Global Accelerator and, somewhat ironically given the context, we decided to deploy it today. Pretty neat! I'll have to keep in mind that I learned about it because of a worldwide outage :) Thanks!

Re: Tell HN: AWS appears to be down again

#475

Post by [deleted] »

dilyevsky wrote: post #315
electroly wrote: post #213

Earlier quoted context omitted.

Multi-region is difficult and expensive, and a lot of projects aren't that important. Most of our infrastructure just isn't that vital; we'd rather take the occasional outage than spend the time and money implementing the sort of active-active multi-region infrastructure that a "correct" implementation would use. We took the recent 8 hour us-east-1 outage on the nose and have not reconsidered this plan. It was a calc…

Curios if you tell your customers you’re totally ok with having lower than 99.9 availability

[deleted]

Re: Tell HN: AWS appears to be down again

#477

Post by dexterdog »

s_dev wrote: post #373
tomlagier wrote: post #347

I wonder if AWS will make more or less money from these outages? Will large players flee because of excessive instability? Or will smaller players go from single-AZ to more expensive multi-AZ? My guess is that no-one will leave and lots of single-AZ tenants who should be multi-AZ will use this as the impetus to do it. Honestly, having events like this is probably good for the overall resilience of distributed systems…

>I wonder if AWS will make more or less money from these outages? There is no possibility that outages are good for AWS. Nor is there more money to be made from "publicity" of the outages.

> There is no possibility that outages are good for AWS.

Do you know how many non-technical CEOs/boards/bosses have told their tech people that they need to go multi-region/cloud because that's what the one-paragraph blog and/or tweet told them to do in response to last weeks event?

Re: Tell HN: AWS appears to be down again

#478

Post by titanomachy »

mikestew wrote: post #344
pier25 wrote: post #299

Earlier quoted context omitted.

Absolutely. My wife and I lived for a year in an off-the-grid cabin in some mountains in Mexico. We had solar panels and a generator we used only when absolutely necessary. We were never without power, but we lived with the constant anxiety of optimizing our energy consumption. Some stuff we could only do during the day and at night we only used devices with batteries. For a couple of weeks we didn't have running wat…

Absolutely. My wife and I lived for a year in an off-the-grid cabin in some mountains in Mexico. I think everyone ought to do a week in an RV with no connections to utilities. Not to take away from your story, but a similar scenario comes up when we "dry camp" (no water or electrical connections): resources are not unlimited. We have solar panels, big-ass inverter and big-ass battery to go with it. But if we want lig…

Or even better, go backpacking in the wilderness. Slightly different set of constraints: you can usually find water (at least where I hike), but carrying all your equipment and food on your back gives you a new perspective on what's "essential".

Re: Tell HN: AWS appears to be down again

#479

Post by mr_toad »

nerdjon wrote: post #136
iamricks wrote: post #116

Earlier quoted context omitted.

lol imagine if azure was just AWS in the backend

Is it bad that I can almost see that being a quick and dirty MVP to get out the door while you built your own cloud solution? Raises serious migration and cost issues, but... would be interesting.

> a quick and dirty MVP to get out the door while you built your own cloud solution?

Seemed to work for Dropbox.

Re: Tell HN: AWS appears to be down again

#480

Post by elasticventures »

staticassertion wrote: post #167
rfoo wrote: post #146

Earlier quoted context omitted.

Satire. Every time Github went down multiple people post on HN saying "every since they were bought by Microsoft, ...". As annoying as those Rust evangelists on every single memory corruption bug.

> As annoying as those Rust evangelists on every single memory corruption bug. First of all, how dare you! Second, shoulda used rust ¯\_(ツ)_/¯

I could have written the OP message a year ago -- I used to feel the same way.

Plz don't disparage Rust evangelism!

Rust is awesome. yes it is complex, frequently annoying, easy to learn difficult to master. I'm speaking from a 30 year dev career.

a few months ago I intended to do a quick investigation into RUST to validate my "i really don't need to learn this" specifically for an embedded project. Within a few hours I found I had become a zealot. Rust has too many "omg, i should tell everybody about this" behaviors that I can't even find my favorite aspect yet.

It's equivalent to a lost soul finding Christianity and accepting the lords blessing and forgiveness! The weight that is lifted of being forgiven to your sins resulting == no more guilt, it's all forgiven! immediately reduction of cognitive dissonance. in this example with rust, it's pointer tracking and memory management, but it's basically the same thing. Rust is for the pious developer.

Those people who are still using C++ for fresh starts are the same folks who love to do things the hard & wrong way, or at least those who don't know any better, infidels, unwashed heathen.

Join us. join rUSt.

Post reply on HN