Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern cloud architecture on AWS: server fleets and databases

start.jcolemorrison.com

Post reply on HN
Print view View original

41–50 of 54 posts

Re: Modern cloud architecture on AWS: server fleets and databases

#41

Post by scarface74 »

malisper wrote: post #37
appwiz wrote: post #35

Earlier quoted context omitted.

You can have millions of consumers read concurrently from a single SQS queue. Messages that are read remain in the queue up to the configured retention period or until a consumer calls DeleteMessage. Source: I’ve built very high volume services that continue to run production workloads and use SQS as the buffer between components.

> You can have millions of consumers read concurrently from a single SQS queue. We're using different definitions of "consumer". By consumer, I'm talking about a group of workers that processes the data for one purpose. For example you may have one consumer read from the queue to generate various metrics and a second consumer read from the queue and write to a DB. With vanilla SQS, when you process a message, you nee…

One producer/multiple consumers is what SNS+ Attributes + subscription filters + SQS is for.

If your database goes down, you have point in time recovery and read replicas that can be promoted as needed.

Re: Modern cloud architecture on AWS: server fleets and databases

#42

Post by viraptor »

scarface74 wrote: post #40
malisper wrote: post #26

Although there are a ton of AWS servers, there's only a few core services that I recommend: EC2 - You need a server. RDS - You need a database. S3 - You need to store files. Lambda - You are building an API with short lived requests. These services are all very high quality and are excellent at what they do. Once you get outside of these core services, the quality quickly drops. You're probably better off using the n…

Why would you ever use Terraform over CloudFormation? There are so many parts of AWS that use CF and that you can modify from the getting started templates like CodeStar and exporting a SAM template from your lambda template. Before someone comments on how TF is “cross platform”, all of the provisioners are vendor specific. As far as what other services to use, if you are hosting your own services on AWS instead of u…

> Why would you ever use Terraform over CloudFormation?

1. You're using Terraform already for resources outside of AWS (cdn, monitoring, dns, anything else) and want to stay with a common tech.

2. You're running into cases that CF doesn't support and have to generate your descriptions externally, or use sparkleformation hacks.

3. You want to manage a new AWS service. (CloudFormation support lags behind Terraform, new services don't get CF resources for months)

Re: Modern cloud architecture on AWS: server fleets and databases

#43

Post by viraptor »

malisper wrote: post #26

Although there are a ton of AWS servers, there's only a few core services that I recommend: EC2 - You need a server. RDS - You need a database. S3 - You need to store files. Lambda - You are building an API with short lived requests. These services are all very high quality and are excellent at what they do. Once you get outside of these core services, the quality quickly drops. You're probably better off using the n…

> you should be using Datadog over CloudWatch

DataDog is great, but the way it polls data means you can't rely on it being available for a long time: https://docs.datadoghq.com/integrations/faq/cloud-metric-del...

> If you receive 1-minute metrics with CloudWatch, then their availability delay is about 2 minutes—so total latency to view your metrics may be ~10-12 minutes.

If an alert delayed by 10min matters to you, DD is not viable for alerting (could be still used for dashboards).

Re: Modern cloud architecture on AWS: server fleets and databases

#44

Post by viraptor »

sciurus wrote: post #31
tmpz22 wrote: post #29

Earlier quoted context omitted.

> Unfortunately there's a lot more too it than that. Is there though? Consider what I would argue to be the "average" case: * Your database never exceeds > 40% resource usage * You service fewer then 1m queries/day * You never burst more then 1k queries/minute * You have a script tied to a cronjob that backs up the database, with basic error handling that sends you a Slack DM if it fails * You have a script tied to a…

Any that can't afford more than a couple minutes of downtime when a server fails.

That's definitely not an "average" company. It's also a really small number of companies that really can't afford that, rather than "earn less money than usual".

Re: Modern cloud architecture on AWS: server fleets and databases

#46

Post by bradknowles »

viraptor wrote: post #43
malisper wrote: post #26

Although there are a ton of AWS servers, there's only a few core services that I recommend: EC2 - You need a server. RDS - You need a database. S3 - You need to store files. Lambda - You are building an API with short lived requests. These services are all very high quality and are excellent at what they do. Once you get outside of these core services, the quality quickly drops. You're probably better off using the n…

> you should be using Datadog over CloudWatch DataDog is great, but the way it polls data means you can't rely on it being available for a long time: https://docs.datadoghq.com/integrations/faq/cloud-metric-del... > If you receive 1-minute metrics with CloudWatch, then their availability delay is about 2 minutes—so total latency to view your metrics may be ~10-12 minutes. If an alert delayed by 10min matters to you,…

CloudWatch Logs has a lot of its own internal latencies. If you can send the logs straight from your ec2 hosts to the log processing system and bypass CloudWatch, then you only want/need CWL for the things you can’t get from those logs.

CloudWatch Metrics is a totally separate beast, which happens to share a similar name. You can set up basic alerts in CWM, and you can trigger certain types of events from those alerts, but it is still very limited. If you want real monitoring and alerting, then CWM isn’t even the easy 80%.

Re: Modern cloud architecture on AWS: server fleets and databases

#47

Post by viraptor »

bradknowles wrote: post #46
viraptor wrote: post #43

Earlier quoted context omitted.

> you should be using Datadog over CloudWatch DataDog is great, but the way it polls data means you can't rely on it being available for a long time: https://docs.datadoghq.com/integrations/faq/cloud-metric-del... > If you receive 1-minute metrics with CloudWatch, then their availability delay is about 2 minutes—so total latency to view your metrics may be ~10-12 minutes. If an alert delayed by 10min matters to you,…

CloudWatch Logs has a lot of its own internal latencies. If you can send the logs straight from your ec2 hosts to the log processing system and bypass CloudWatch, then you only want/need CWL for the things you can’t get from those logs. CloudWatch Metrics is a totally separate beast, which happens to share a similar name. You can set up basic alerts in CWM, and you can trigger certain types of events from those alert…

Agreed, my message was about using DD instead of CW for internal logs. With your own logs, you've got so much more flexibility - but not everything can be done that way. For example ELB stats don't really exist elsewhere in realtime.

Re: Modern cloud architecture on AWS: server fleets and databases

#48

Post by bradknowles »

nunez wrote: post #22
Jonnax wrote: post #4

At what scale would you want to use RDS rather than using an EC2 instance with Postgres installed? Assuming that the operator has the skills to manage Postgres. It's not like RDS does something complex like Geodistribution, right? Also what is the scaling like? Is it automatic? How quickly can you handle more connections? Because my understanding was that it was slow. I did have a play with their RDS Postgres nonths…

You would move to RDS the minute you know you're committed to AWS and know that you don't want to worry about things like backups, upgrades or clustering. The disadvantage to doing so is that you lose a lot of administrative privileges to the database server itself and you don't get access to the filesystem or underlying OS. We had trouble migrating a sizable SQL Server installation onto RDS because it had a ton of t…

Yeah, well. That’s SQL Server for you. There’s a lot of things that RDS can’t do for you in the SQL Server flavor.

RDS MySQL != RDS Postgres && RDS (MySQL || Postgres) >> RDS SQL Server

Re: Modern cloud architecture on AWS: server fleets and databases

#49

Post by BrentOzar »

kamilafsar wrote: post #2

I keep reading all these horror stories about Aurora (especially PostgreSQL). Is there anyone out there with an alternative story?

We run a 2TB database with 30-60 days of data (only keeps 30-60 days on-hand, and we're roughly ingesting 50GB/day.) We've been using Aurora Postgres since it came out, and it's been pretty good. (Good enough to the point that it's never crossed my mind to think about moving to something else.)

Source: Microsoft SQL Server performance tuner who runs an app that centralizes SQL Server performance data from thousands of servers. You'd think I would be running MSSQL on the back end, but the licensing costs just didn't make sense compared to Aurora Postgres.

Re: Modern cloud architecture on AWS: server fleets and databases

#50

Post by nostrebored »

CoolGuySteve wrote: post #34
reilly3000 wrote: post #23

Earlier quoted context omitted.

RDS can scale read replicas and fail over to master, but aurora and it’s serverless option is much better for auto-scaling. Behind the scenes storage is decoupled from compute with makes scaling fundamentally easier. EC2 is your only choice if you want a database that AWS doesn’t support, such as Rethink or Cassandra (they just recently launched a managed Cassandra service though). EC2 is also your only choice if you…

I went with Aurora Postgres recently for a new application and to be honest it fucking sucks. When it scales up, queries start to fail and there are long delays. I had to add a try/retry loop around all my inserts to avoid losing data. I definitely wouldn't recommend it for bursty applications where it needs to rapidly scale up. The thing that bugs me is that I know a similar workload works fine on a relatively cheap…

... were you not handling errors on database inserts through well established concepts like exponential backoff and retry already?

that seems like a colossally bad idea

If you don't need the striped storage model and guarantees that Aurora provides, it might not be the service for you

Post reply on HN