Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

Post reply on HN
Print view View original

141–150 of 153 posts

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#141

Post by roman_sf »

otterley wrote: post #128
roman_sf wrote: post #116

Earlier quoted context omitted.

What do you mean by cross-regional dependencies? Isn't running in multi-region setup is by itself adding dependency? Speaking about multi-region services. What do you think about Google now offering all three major building pieces as multi-regional? They have muti-regional buckets, LB with single anycast IP, document db (firebase). Pubsub can route automatically to nearest region. Nothing like this is available in am…

If your workload can run in region B even if there is a serious failure of a service in region A, in which your workload normally runs, then no, you have not created a cross-regional dependency. When I talk about cross regional dependency, I talk about an architectural decision that can lead to a cascading failure in region B, which is healthy by all accounts, when there is a failure in region A. AWS has services tha…

I haven't used global accelerator but it doesn't look like the same. On landing page it says: "Your traffic routing is managed manually, or in console with endpoint traffic dials and weights".

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#142

Post by otterley »

roman_sf wrote: post #141
otterley wrote: post #128

Earlier quoted context omitted.

If your workload can run in region B even if there is a serious failure of a service in region A, in which your workload normally runs, then no, you have not created a cross-regional dependency. When I talk about cross regional dependency, I talk about an architectural decision that can lead to a cascading failure in region B, which is healthy by all accounts, when there is a failure in region A. AWS has services tha…

I haven't used global accelerator but it doesn't look like the same. On landing page it says: "Your traffic routing is managed manually, or in console with endpoint traffic dials and weights".

“Global Accelerator continuously monitors the health of all endpoints. When it determines that an active endpoint is unhealthy, Global Accelerator instantly begins directing traffic to another available endpoint. This allows you to create a high-availability architecture for your applications on AWS.”

https://docs.aws.amazon.com/global-accelerator/latest/dg/dis...

Alternatively, global load balancing with Route 53 remains a viable, mature option as well. Health checks and failover are fully supported.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#143

Post by otterley »

Corrado wrote: post #134
otterley wrote: post #96

Earlier quoted context omitted.

(Disclaimer: I work for AWS but opinions are my own. I also do not work with the Kinesis team.) Nearly all AWS services are regional in scope, and for many (if not most) services, they are scaled at a cellular level within a region. Accounts are assigned to specific cells within that region. There are very, very few services that are global in scope, and it is strongly discouraged to create cross-regional dependencie…

This outage highlighted our dependency on Cognito. Everything else we are doing can (and probably should) be replicated to another region, which would resolve these types of issues. However, Cognito is very region specific and there is currently no way to run in active-active or even in standby mode. The problem is user accounts; you can't sync them to another region and you can't back-up/restore them (with passwords…

Please bring this to the attention of your account team! They will bring your feedback to the service team. While I can’t speak for the Cognito team, I can assure you they care deeply about customer satisfaction.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#144

Post by freehunter »

qz2 wrote: post #50
freehunter wrote: post #44

Earlier quoted context omitted.

>you can’t trust one provider at all The hard part with multi-cloud is, you're just increasing your risk of being impacted by someone's failure. Sure if you're all-in on AWS and AWS goes down, you're all-out. But if you're on [AWS, GCP] and GCP goes down, you're down anyway. Even though AWS is up, you're down because Google went down. And if you're on [AWS, GCP, Azure] and Azure goes down, it doesn't matter than AWS…

I disagree. It’s about mitigating the risk of a single provider’s failure. Single providers go down all the time. We’ve seen it from all three major cloud vendors.

You disagree with what? That relying on three vendors increases your risk of being impacted by one? That's just statistics. You can disagree with it, but that doesn't make it incorrect.

Or do you disagree that planning for a total failure of one and running redundant workloads on other vendors increases your costs 99.99999% of the time? Because that's a fairly standard SLA from each of the major vendors. Let's even reduce it to EC2's SLA, 99.99%. So 99.99% of the time you're paying 3x as much as you need to be paying just to maintain your services an extra four hours per year. Again, you can disagree with that but that doesn't make it incorrect.

Some businesses might need that extra four hours, the cost of the extra services might be cheaper than the cost of four hours of downtime per year. But you're not going to find many businesses like that. Either you're running completely redundant workloads, paying 3x as much for an extra 4 hours per year, or you're going to be taken offline when any one of the three go down independently of each other.

Single providers go down, yes. And three providers go down three times as often as one. Either you're massively overspending or you're tripling your risk of downtime. If multi-cloud worked, you'd be hearing people talking about it and their success stories would fill the front page of Hacker News. They don't, because it doesn't.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#145

Post by joneholland »

foota wrote: post #140
joneholland wrote: post #122

Earlier quoted context omitted.

Any time you traverse a firewall or a NAT you’ll run into issues if you have a massive amount of open sessions and you are dealing with a stateful FW. Just because you haven’t encountered it doesn’t mean it’s not there, it’s probably just properly tuned and balanced for the load.

Yeah, I believe that. I'm not sure this applies to something like AWS though, where firewall like capability is provided via a layer spread over thousands or greater of instances.

The virtual firewalls running on the virtual networks in AWS for their customers are not the same as the layer 2 firewalls that exist in their data centers internally.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#146

Post by cowsandmilk »

freehunter wrote: post #44
qz2 wrote: post #16

Earlier quoted context omitted.

Correct. I, as many people have, discovered this when something broke in one of the golden regions. In my case cloudfront and ACM. Realistically you can’t trust one provider at all if you have high availability requirements. The justification is apparently that the cloud is taking all this responsibility away from people but from personal experience running two cages of kit at two datacenters the TCO was lower and th…

>you can’t trust one provider at all The hard part with multi-cloud is, you're just increasing your risk of being impacted by someone's failure. Sure if you're all-in on AWS and AWS goes down, you're all-out. But if you're on [AWS, GCP] and GCP goes down, you're down anyway. Even though AWS is up, you're down because Google went down. And if you're on [AWS, GCP, Azure] and Azure goes down, it doesn't matter than AWS…

> which means you're paying 3x more than you need to 99.99999% of the time.

This would be a poor decision. If you assume AWS, GCP, and Azure would fail independently, you can pay 1.5x. Each of the 3 services would be scaled to take 50% of your traffic. If any one fails, you would then still be able to handle 100%. This is a common way to structure applications. Assuming independence means that more replicas result in less overprovisioning. 1 replica means needing to provision 2x. Having 5 independent replicas means, you need to provision 1.25x to be resilient against one failure as each replica will be scaled at 25%.

In general, N replicas need N/(N-1) over provisioning to be resilient against one replica failing.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#147

Post by akhilcacharya »

amf12 wrote: post #139
marcinzm wrote: post #29

Earlier quoted context omitted.

I don't think it's about growing fast so much as, from those I talked to, Amazon now has a fairly bad reputation in the tech community. You only go to work there if you don't have a better option (Google, Facebook, etc) or have some specialty skill they're willing to pay for. Pay is below other FAANG companies and the work culture isn't great (toxic even some would say). edit: They also had the most disorganized and…

> I don't think it's about growing fast so much as, from those I talked to, Amazon now has a fairly bad reputation in the tech community. My personal observation having known quite a few Amazon SWEs and interviewed them. The bad rep is only for the junior roles. SWEs who work at AWS and are high L5+ are pretty solid.

What exactly does high L5+ mean?

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#148

Post by amf12 »

akhilcacharya wrote: post #147
amf12 wrote: post #139

Earlier quoted context omitted.

> I don't think it's about growing fast so much as, from those I talked to, Amazon now has a fairly bad reputation in the tech community. My personal observation having known quite a few Amazon SWEs and interviewed them. The bad rep is only for the junior roles. SWEs who work at AWS and are high L5+ are pretty solid.

What exactly does high L5+ mean?

SWEs who weren't just recently promoted to L5 at Amazon. They have some experience at that level. Granted, there could be some bias because it's not easy to pinpoint when they were promoted.

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#149

Post by akhilcacharya »

amf12 wrote: post #148
akhilcacharya wrote: post #147

Earlier quoted context omitted.

What exactly does high L5+ mean?

SWEs who weren't just recently promoted to L5 at Amazon. They have some experience at that level. Granted, there could be some bias because it's not easy to pinpoint when they were promoted.

As a recently promoted L5, what bad rep should I be shielding myself from as a junior?

Re: Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region

#150

Post by Corrado »

otterley wrote: post #143
Corrado wrote: post #134

Earlier quoted context omitted.

This outage highlighted our dependency on Cognito. Everything else we are doing can (and probably should) be replicated to another region, which would resolve these types of issues. However, Cognito is very region specific and there is currently no way to run in active-active or even in standby mode. The problem is user accounts; you can't sync them to another region and you can't back-up/restore them (with passwords…

Please bring this to the attention of your account team! They will bring your feedback to the service team. While I can’t speak for the Cognito team, I can assure you they care deeply about customer satisfaction.

That's a great idea. I'm writing an email right now! :)
Post reply on HN