Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern cloud architecture on AWS: server fleets and databases

start.jcolemorrison.com

Post reply on HN
Print view View original

31–40 of 54 posts

Re: Modern cloud architecture on AWS: server fleets and databases

#31

Post by sciurus »

tmpz22 wrote: post #29
malisper wrote: post #27

Earlier quoted context omitted.

> In a couple hours you should be able to setup automatic backups and practice going through the recover process a couple times. Unfortunately there's a lot more too it than that. You need to handle when the backup job fails or dies, have a process for deleting old backups, etc. Not just that, but if you have multiple Postgres instances, you need to do this work for each machine. I've seen first hand this kind of stu…

> Unfortunately there's a lot more too it than that. Is there though? Consider what I would argue to be the "average" case: * Your database never exceeds > 40% resource usage * You service fewer then 1m queries/day * You never burst more then 1k queries/minute * You have a script tied to a cronjob that backs up the database, with basic error handling that sends you a Slack DM if it fails * You have a script tied to a…

Any that can't afford more than a couple minutes of downtime when a server fails.

Re: Modern cloud architecture on AWS: server fleets and databases

#32

Post by mdeeks »

nunez wrote: post #24
mdeeks wrote: post #21

Earlier quoted context omitted.

I'll take a shot at this. There is always an asterisk under every one of these. Every company and situation is different. If you're a tiny startup or hobby with literally no money, it might make sense for you to manage it yourself because you have no choice. Once you have some money and a viable business, then your value is no longer your ability to spend your time running Postgres, ensuring backups and restores work…

If I were a "tiny startup", I would absolutely leverage RDS in the first instance. Last thing I want to spend time doing is setting us Postgres and accumulating tech debt from a bad or suboptimal install. There are RDS instances that qualify for Free Tier as well if you're running _really_ lean. In fact, the only time that I would consider migrating to a DB on EC2 is if the database isn't supported by RDS (DB2, for e…

The key words were "literally no money". By tiny startup I meant one or two people without any funding, revenue, and saddled with life expenses like rent, food, college savings, etc.

I definitely agree that a DB on EC2 is tech debt. But sometimes that is something you have to accept for immediate financial reasons while you figure out how to create a product with revenue to cover costs.

Re: Modern cloud architecture on AWS: server fleets and databases

#33

Post by CoolGuySteve »

malisper wrote: post #30
reese_john wrote: post #28

Earlier quoted context omitted.

Just curious, why would you not recommend SQS ?

I've never used SQS but IMO it seems inferior to Kinesis or Kafka. The two big reasons are that you can't have multiple consumers read from a single queue and once data leaves the queue, it's gone forever. Both Kinesis and Kafka let you have multiple consumers and configure a retention period for your messages.

How so? I've run thousands of consumers on SQS for batch jobs and it seems to work.

There's also dead letter queue and retries for messages that aren't properly serviced.

Re: Modern cloud architecture on AWS: server fleets and databases

#34

Post by CoolGuySteve »

reilly3000 wrote: post #23
Jonnax wrote: post #4

At what scale would you want to use RDS rather than using an EC2 instance with Postgres installed? Assuming that the operator has the skills to manage Postgres. It's not like RDS does something complex like Geodistribution, right? Also what is the scaling like? Is it automatic? How quickly can you handle more connections? Because my understanding was that it was slow. I did have a play with their RDS Postgres nonths…

RDS can scale read replicas and fail over to master, but aurora and it’s serverless option is much better for auto-scaling. Behind the scenes storage is decoupled from compute with makes scaling fundamentally easier. EC2 is your only choice if you want a database that AWS doesn’t support, such as Rethink or Cassandra (they just recently launched a managed Cassandra service though). EC2 is also your only choice if you…

I went with Aurora Postgres recently for a new application and to be honest it fucking sucks.

When it scales up, queries start to fail and there are long delays. I had to add a try/retry loop around all my inserts to avoid losing data.

I definitely wouldn't recommend it for bursty applications where it needs to rapidly scale up.

The thing that bugs me is that I know a similar workload works fine on a relatively cheap c3.xlarge so I'm not actually saving much money with Aurora.

Re: Modern cloud architecture on AWS: server fleets and databases

#35

Post by appwiz »

malisper wrote: post #30
reese_john wrote: post #28

Earlier quoted context omitted.

Just curious, why would you not recommend SQS ?

I've never used SQS but IMO it seems inferior to Kinesis or Kafka. The two big reasons are that you can't have multiple consumers read from a single queue and once data leaves the queue, it's gone forever. Both Kinesis and Kafka let you have multiple consumers and configure a retention period for your messages.

You can have millions of consumers read concurrently from a single SQS queue. Messages that are read remain in the queue up to the configured retention period or until a consumer calls DeleteMessage.

Source: I’ve built very high volume services that continue to run production workloads and use SQS as the buffer between components.

Re: Modern cloud architecture on AWS: server fleets and databases

#36

Post by fragmede »

malisper wrote: post #26

Although there are a ton of AWS servers, there's only a few core services that I recommend: EC2 - You need a server. RDS - You need a database. S3 - You need to store files. Lambda - You are building an API with short lived requests. These services are all very high quality and are excellent at what they do. Once you get outside of these core services, the quality quickly drops. You're probably better off using the n…

Depending on the market segment you exist in, I'd recommend AWS Fargate and AWS Lightsail (container-runner; Digital Ocean/Linode/VPS competitor) over EC2. There's absolutely a segment for which EC2 is appropriate, but just like most data isn't "big", I doubt that most EC2 customers wouldn't be better served by Lightsail. If you've got several hundred or several thousand EC2 instances with bespoke code/config for many different ASGs, then Lightsail isn't for you, but (my impression is) that's not most people.

Re: Modern cloud architecture on AWS: server fleets and databases

#37

Post by malisper »

appwiz wrote: post #35
malisper wrote: post #30

Earlier quoted context omitted.

I've never used SQS but IMO it seems inferior to Kinesis or Kafka. The two big reasons are that you can't have multiple consumers read from a single queue and once data leaves the queue, it's gone forever. Both Kinesis and Kafka let you have multiple consumers and configure a retention period for your messages.

You can have millions of consumers read concurrently from a single SQS queue. Messages that are read remain in the queue up to the configured retention period or until a consumer calls DeleteMessage. Source: I’ve built very high volume services that continue to run production workloads and use SQS as the buffer between components.

> You can have millions of consumers read concurrently from a single SQS queue.

We're using different definitions of "consumer". By consumer, I'm talking about a group of workers that processes the data for one purpose. For example you may have one consumer read from the queue to generate various metrics and a second consumer read from the queue and write to a DB. With vanilla SQS, when you process a message, you need to perform all the tasks simultaneously. With Kinesis and Kafka you can have independent groups of workers (i.e. independent consumers), each performing one of these tasks. Each consumer is able to process the queue at it's own rate. The way Amazon recommends doing this in SQS is to have SNS fan out a single SQS queue to multiple SQS queues. Then you can consume each queue independently[0]. That will multiply your costs by the number of queues you have.

> Messages that are read remain in the queue up to the configured retention period or until a consumer calls DeleteMessage.

I'm talking about retaining a message even if it was successfully processed, on the order of days or weeks. I've used this feature of Kafka before to implement a recovery log. Under normal operation, Kafka writes data to a DB. If the DB goes down, you can quickly recover the last N days of data by going through the data retained in Kafka.

[0] https://forums.aws.amazon.com/message.jspa?messageID=865925

Re: Modern cloud architecture on AWS: server fleets and databases

#38

Post by malisper »

CoolGuySteve wrote: post #33
malisper wrote: post #30

Earlier quoted context omitted.

I've never used SQS but IMO it seems inferior to Kinesis or Kafka. The two big reasons are that you can't have multiple consumers read from a single queue and once data leaves the queue, it's gone forever. Both Kinesis and Kafka let you have multiple consumers and configure a retention period for your messages.

How so? I've run thousands of consumers on SQS for batch jobs and it seems to work. There's also dead letter queue and retries for messages that aren't properly serviced.

See my reply to appwiz: https://news.ycombinator.com/item?id=22280200

Re: Modern cloud architecture on AWS: server fleets and databases

#39

Post by fragmede »

tmpz22 wrote: post #29
malisper wrote: post #27

Earlier quoted context omitted.

> In a couple hours you should be able to setup automatic backups and practice going through the recover process a couple times. Unfortunately there's a lot more too it than that. You need to handle when the backup job fails or dies, have a process for deleting old backups, etc. Not just that, but if you have multiple Postgres instances, you need to do this work for each machine. I've seen first hand this kind of stu…

> Unfortunately there's a lot more too it than that. Is there though? Consider what I would argue to be the "average" case: * Your database never exceeds > 40% resource usage * You service fewer then 1m queries/day * You never burst more then 1k queries/minute * You have a script tied to a cronjob that backs up the database, with basic error handling that sends you a Slack DM if it fails * You have a script tied to a…

> * Your database never exceeds > 40% resource usage > * You service fewer then 1m queries/day > * You never burst more then 1k queries/minute

How do I know it doesn't exceed 40% usage? Better yet, who's holding the pager when it does? If/when it does, who's product launch is dead in the water while the db is reconfigured onto a larger instance? What product isn't being delivered because we're faffing about with the database instead of product code?

> * You have a script tied to a cronjob that backs up the database, with basic error handling that sends you a Slack DM if it fails > * You have a script tied to a cronjob which deletes old backups, with basic error handling that sends you a Slack DM if it fails

Who's responsible for restoring from backup every week/month/quarter, to assert they actually work, with whatever changes have been made recently? Untested backups are Shrodingers backups.

Just how well tested is this script? Does it properly error out if the script fails to be run? What if a firewall rule accidentally gets set that blocks egress from the backup box to the Internet (for security); who/how/what gets notified instead? Who's deliverables are slipping because the backups randomly stopped working?

> What percentage of companies need more then that?

That's a fair question, but Amazon's done far more research than I, possibly you on that topic. The real question is, of companies that don't need more than that, how many companies want to hire somebody to take on those responsibilities part-time? How many companies have the expertise to even hire somebody qualified to do that part-time? And since those people are managing the DB part time, how many of them are giving it the attention it needs, and aren't distracted by other responsibilities to the company?

None of those problems are insurmountable, but they're far from most business' core competency, and time I'm spending dealing with postgresql.conf (or my.cnf) is time I'm not dealing with other issues. Don't get me wrong, there's still a time and place for managing database instances, but IMO small business (small > tiny) aren't the appropriate place for that. I'd be interested in hearing if someone's run the numbers to justify it though! (Especially if it falls in favor of running it yourself.)

Re: Modern cloud architecture on AWS: server fleets and databases

#40

Post by scarface74 »

malisper wrote: post #26

Although there are a ton of AWS servers, there's only a few core services that I recommend: EC2 - You need a server. RDS - You need a database. S3 - You need to store files. Lambda - You are building an API with short lived requests. These services are all very high quality and are excellent at what they do. Once you get outside of these core services, the quality quickly drops. You're probably better off using the n…

Why would you ever use Terraform over CloudFormation? There are so many parts of AWS that use CF and that you can modify from the getting started templates like CodeStar and exporting a SAM template from your lambda template.

Before someone comments on how TF is “cross platform”, all of the provisioners are vendor specific.

As far as what other services to use, if you are hosting your own services on AWS instead of using AWS manager services, you’re kind of missing the point of AWS.

But a few other services we use all of the time are CodeBuild, ElasticCache (hosted Redis), ElasticSearch, Route 53, load balancers, autoscaling groups, SSM (managing the few “pets” until we can kill them), ECS, ECR, Fargate, SNS, SQS, DynamoDB, SFTP, CloudTrail, Microsoft AD, we are experimenting with the recently announced Device Farm/Selenium service, step functions, Athena, Secrets Manager, and a few more I’m probably forgetting.

Post reply on HN