Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

Post reply on HN
Print view View original

91–100 of 153 posts

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#91

Post by edoceo »

Twirrim wrote: post #69
edoceo wrote: post #47

Earlier quoted context omitted.

Oof. My little company is refactoring some five year old architecture design choices. Ugly. Process isn't visible outside the refactor and the work is tedious. Can't imagine what a service refactor is like at A. I bet it sucks

> Can't imagine what a service refactor is like at A. I bet it sucks It's not all that hard. AWS heavily focuses on Service Oriented Architecture approaches, with specific knowledge/responsibility domains for each. It's a proven scalable pattern. The APIs will often be fairly straight-forward behind the front end. With clearly lines of responsibility between components, you'll almost never have to worry about what ot…

Your side note should be the first point.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#92

Post by WookieRushing »

londons_explore wrote: post #79

One requirement on my "production ready" checklist is that any catastrophic system failure can be resolved by starting a completely new instance of the service, and it be ready to serve traffic inside 10 minutes. That should be tested at least quarterly (but preferably automatically with every build). If Amazon did that, this outage would have been reduced to 10 mins, rather than the 12+ hours that some super slow ro…

This only works for stateless services. If you’ve got frontends that take longer than 10 mins to serve traffic then you have a problem.

But if you’re running a DB or a storage system, 10 mins is a blink of an eye. Storage systems in particular can run a few hundred TB per node and moving that data to another node can take over an hour.

In this case, the frontends have a shard map which is definitely not stateless. This is typically okay if you have a fast load operation which blocks other traffic until shard map is fully loaded

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#93

Post by ipsocannibal »

So the cause of outage boils down to not having a metric on total file descriptors with an alarm if usage gets within 10% of the Max and a faulty scaling plan that should of said "for every N backend hosts we add we must add X frontend hosts". One metric and a couple of lines in a wiki could have saved Amazon what is probably millions in outage related costs. One wonders if Amazon retail will start hedging its bets and go multicloud to prevent impacts on the retail customers from AWS LSE's.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#94

Post by crgwbr »

ryanworl wrote: post #76
justicezyx wrote: post #72

Earlier quoted context omitted.

I dont remember the size, is it 128bits? It was chosen for future expansion. Kinesis was envisioned to be a much larger-scale Kafka + Storm (storm was the streaming programming framework popular in 2012, it was since falls out of favor).

128bit might be accurate, I meant more along the lines of they are non-contiguous and don't seem to be correlated with the amount of records actually being written to a stream.

I always assumed they were sequential per-region rather than per-stream, but that’s just a guess.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#95

Post by coredog64 »

qz2 wrote: post #16
EdwardDiego wrote: post #10

Earlier quoted context omitted.

Indeed. We're paying (and designing our systems to work on multiple AZs) to reduce the risk of outages, but then their back-end services are reliant on services in a sole region?

Correct. I, as many people have, discovered this when something broke in one of the golden regions. In my case cloudfront and ACM. Realistically you can’t trust one provider at all if you have high availability requirements. The justification is apparently that the cloud is taking all this responsibility away from people but from personal experience running two cages of kit at two datacenters the TCO was lower and th…

ACM and CloudFront being sticky to us-east-1 is particularly annoying. I’m happy not being multi regional (I don’t have that level of DR requirements), but these types of services require me to incorporate all the multi region complexity.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#96

Post by otterley »

EdwardDiego wrote: post #10
ris wrote: post #8

The one thing I want to know in cases like this is: why did it affect multiple Availability Zones? Making a resource multi-AZ is a significant additional cost (and often involves additional complexity) and we really need to be confident that typical observed outages would actually have been mitigated in return.

Indeed. We're paying (and designing our systems to work on multiple AZs) to reduce the risk of outages, but then their back-end services are reliant on services in a sole region?

(Disclaimer: I work for AWS but opinions are my own. I also do not work with the Kinesis team.)

Nearly all AWS services are regional in scope, and for many (if not most) services, they are scaled at a cellular level within a region. Accounts are assigned to specific cells within that region.

There are very, very few services that are global in scope, and it is strongly discouraged to create cross-regional dependencies -- not just as applied to our customers, but to ourselves as well. IAM and Route 53 are notable exceptions, but they offer read replicas in every region and are eventually consistent: if the primary region has a failure, you might not be able to make changes to your configuration, but the other regions will operate on read-only replicas.

This incident was regional in scope: us-east-1 was the only impacted region. As far as I know, no other region was impacted by this event. So customers operating in other regions were largely unaffected. (If you know otherwise, please correct me.)

As a Solutions Architect, I regularly warn customers that running in multiple Availability Zones is not enough. Availability Zones protect you from many kinds of physical infrastructure failures, but not necessarily from regional service failures. So it is super important to run in multiple regions as well: not necessarily active-active, but at least in a standby mode (i.e. "pilot light") so that customers can shed traffic from the failing region and continue to run their workloads.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#97

Post by xiwenc »

Twirrim wrote: post #69
edoceo wrote: post #47

Earlier quoted context omitted.

Oof. My little company is refactoring some five year old architecture design choices. Ugly. Process isn't visible outside the refactor and the work is tedious. Can't imagine what a service refactor is like at A. I bet it sucks

> Can't imagine what a service refactor is like at A. I bet it sucks It's not all that hard. AWS heavily focuses on Service Oriented Architecture approaches, with specific knowledge/responsibility domains for each. It's a proven scalable pattern. The APIs will often be fairly straight-forward behind the front end. With clearly lines of responsibility between components, you'll almost never have to worry about what ot…

Indeed your side note should be the first point. I see that very often in practice. As a result, products get rewritten all the time because developers dont want to spend time understanding the current system believing they can do better job than the previous one. They will create different problems. And the cycle repeats.

The not-invented-here (or by-me) syndrome is probably also at play here.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#98

Post by WJW »

londons_explore wrote: post #79

One requirement on my "production ready" checklist is that any catastrophic system failure can be resolved by starting a completely new instance of the service, and it be ready to serve traffic inside 10 minutes. That should be tested at least quarterly (but preferably automatically with every build). If Amazon did that, this outage would have been reduced to 10 mins, rather than the 12+ hours that some super slow ro…

Kinesis probably runs well over 100k instances. Restarting it might not be so trivial that you can do it in 10 minutes.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#99

Post by londons_explore »

WookieRushing wrote: post #92
londons_explore wrote: post #79

One requirement on my "production ready" checklist is that any catastrophic system failure can be resolved by starting a completely new instance of the service, and it be ready to serve traffic inside 10 minutes. That should be tested at least quarterly (but preferably automatically with every build). If Amazon did that, this outage would have been reduced to 10 mins, rather than the 12+ hours that some super slow ro…

This only works for stateless services. If you’ve got frontends that take longer than 10 mins to serve traffic then you have a problem. But if you’re running a DB or a storage system, 10 mins is a blink of an eye. Storage systems in particular can run a few hundred TB per node and moving that data to another node can take over an hour. In this case, the frontends have a shard map which is definitely not stateless. Th…

It's possible (albeit much harder) for stateful services too.

It basically boils down to "We must be able to restore the minimum necessary parts of a full backup in under 10 minutes".

Take wikipedia as an example. I'd expect them to be able to restore a backup of the latest version of all pages in 10 minutes. It's 20GB of data, and I assume it's sharded at least 10 ways. That means each instance will have to grab 2GB from the backups. Very do-able.

As a service gets bigger, you typically scale horizontally, so the problem doesn't get harder.

Restoring all the old page versions and re enabling editing might take longer, but that's less critical functionality.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#100

Post by nixass »

marcinzm wrote: post #29
joneholland wrote: post #27

Running out of file handles and other IO limits is embarrassing and happens at every company, but I’m surprised that AWS was not monitoring this. I’m also surprised at the general architecture of Kinesis. What appears to be their own hand rolled gossip protocol (that is clearly terrible compared to raft or paxos, a thread per cluster member? Everyone talking to everyone? An hour to reach consensus?) and the front end…

I don't think it's about growing fast so much as, from those I talked to, Amazon now has a fairly bad reputation in the tech community. You only go to work there if you don't have a better option (Google, Facebook, etc) or have some specialty skill they're willing to pay for. Pay is below other FAANG companies and the work culture isn't great (toxic even some would say). edit: They also had the most disorganized and…

Very anecdotal
Post reply on HN