Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS RDS Postmortem: Is AWS Collapsing Under its Own Weight ?

news.ycombinator.com

Post reply on HN
Print view View original

11–19 of 19 posts

Re: AWS RDS Postmortem: Is AWS Collapsing Under its Own Weight ?

#11

Post by ruben81adelaide »

evil-olive wrote: post #8

is this RDS Aurora or RDS "classic"? classic RDS is essentially control-plane-only - AWS spins up an EC2 instance on your behalf, installs the database software, configures replication, etc etc. but on the data plane, you're connecting to a more or less stock Postgres or MySQL instance. Aurora uses a more modern distributed design [0] akin to Spanner, CockroachDB, etc. they implemented their own quorum-based log laye…

@evil-olive

Thanks for your feedback. Aurora is something that we would definitely look in the future, but with the levels of workload that we are currently experiencing postgres RDS should work for quite a long time.

> CockroachDB or Cassandra or something similar.

I can't wait for a business use case where I need something like this :) but unfortunately right now this is not the case.

Re: AWS RDS Postmortem: Is AWS Collapsing Under its Own Weight ?

#12

Post by viraptor »

ruben81adelaide wrote: post #6
ruben81adelaide wrote: post #5

Earlier quoted context omitted.

@synicalx thanks for your feedback. I will see if there is anything visible in Cloud Watch. > - Other than the (presumably) one change made to that RDS instance at ~4.30AM, were there any other changes made to it, specifically to its storage, prior to the autoscaling event? At 4.30 the following has been logged: "Storage size 999 GB is approaching the maximum storage threshold 1000 GB. Increase the maximum storage th…

> Does Cloudwatch confirm that RDS "needed" to scale? Actually you pointed out to a clue that I missed. I should have checked cloud watch! The free space graphs shows that at some point something got the 200 GB of space that we originally assigned. This is a good clue that I am going to dig in. Thanks for your feedback! EDIT: Cloudwatch metrics in RDS has been the key to find the source of the issue.

So what was it? My money would be on (in order): Wal, logs, unexpected indices growth.

Re: AWS RDS Postmortem: Is AWS Collapsing Under its Own Weight ?

#13

Post by fxtentacle »

To me, this reads like they are trying to tell you to get lost in nice words. Most likely, you're not spending $1+ mio annually so they just don't care about you or your experience.

AWS is built to support the biggest cloud empires on the planet. If you're not working on that scale, a smaller provider will likely provide much more personal attention, so better support, better tuning, and potentially better bang per buck.

But I wouldn't call AWS collapsing. It's operating as designed. The issue is that too many small companies bought into their "stand on the shoulders of giants" marketing and then convinced themselves that they need planet-scale whatever when really they don't. If you're small and nimble, you want small and nimble solutions, too.

Re: AWS RDS Postmortem: Is AWS Collapsing Under its Own Weight ?

#14

Post by ruben81adelaide »

fxtentacle wrote: post #13

To me, this reads like they are trying to tell you to get lost in nice words. Most likely, you're not spending $1+ mio annually so they just don't care about you or your experience. AWS is built to support the biggest cloud empires on the planet. If you're not working on that scale, a smaller provider will likely provide much more personal attention, so better support, better tuning, and potentially better bang per b…

@fxtentacle

I agree with you. This actually makes a lot of sense. Probably the title reflected the anger and frustration that I was feeling after the poor support.

Actually the company that is being migrated to AWS is quite large, but not large enough to get in spending $1+mio annually. This migration was one system from many others that needs to be moved.

But, still, 10+ years ago the support in AWS was available for free to everybody, you would get in contact with a big nerd that knew the ins and outs of the system and would help you massively on pretty much anything.

Slowly, they allow just paid support, and now not even paying listen to your problems.

At the very beginning, it was the small(-ish) companies (which a few exceptions) which gave name to AWS, and an army of geeks talking all positive about the services.

If now they just work with large corporations, it is a big (and risky) change, which goes against the Amazon Leadership Principle "Customer Obsession".

Re: AWS RDS Postmortem: Is AWS Collapsing Under its Own Weight ?

#15

Post by ruben81adelaide »

viraptor wrote: post #12
ruben81adelaide wrote: post #6

Earlier quoted context omitted.

> Does Cloudwatch confirm that RDS "needed" to scale? Actually you pointed out to a clue that I missed. I should have checked cloud watch! The free space graphs shows that at some point something got the 200 GB of space that we originally assigned. This is a good clue that I am going to dig in. Thanks for your feedback! EDIT: Cloudwatch metrics in RDS has been the key to find the source of the issue.

So what was it? My money would be on (in order): Wal, logs, unexpected indices growth.

The things started to go from bad to worse when work_mem parameter has not been set up correctly.

Some queries that requires a large amount of memory to process started to use disk.

Once the autoscaling kicked in, even if we would have realized about it, wouldn't have helped as you are locked out of the system.

The auto-scaling event was triggered earlier than our alerts for low disk available.

What I don't know yet, is that we have been moving services for multiple days, and the service that required the work_mem parameter was in production in that AWS for 48 hours before started to use disk rather than memory to process the SQL Queries.

Re: AWS RDS Postmortem: Is AWS Collapsing Under its Own Weight ?

#16

Post by rad_gruchalski »

ruben81adelaide wrote: post #11
evil-olive wrote: post #8

is this RDS Aurora or RDS "classic"? classic RDS is essentially control-plane-only - AWS spins up an EC2 instance on your behalf, installs the database software, configures replication, etc etc. but on the data plane, you're connecting to a more or less stock Postgres or MySQL instance. Aurora uses a more modern distributed design [0] akin to Spanner, CockroachDB, etc. they implemented their own quorum-based log laye…

@evil-olive Thanks for your feedback. Aurora is something that we would definitely look in the future, but with the levels of workload that we are currently experiencing postgres RDS should work for quite a long time. > CockroachDB or Cassandra or something similar. I can't wait for a business use case where I need something like this :) but unfortunately right now this is not the case.

Have a look at YugabyteDB. If you can tolerate Postgres 11.2, it would work for you. Yugabyte offers a managed service, on premise and you can roll out your own system.

I’m not affiliated with the company.

Re: AWS RDS Postmortem: Is AWS Collapsing Under its Own Weight ?

#18

Post by singron »

It's very difficult to manage IOPS for RDS. IOPS scales with RDS disk size to a certain point, so you might not have had nearly enough IOPS at only 200GB. It's possible that you used too small of an instance type since you should probably use an instance that can keep all 15GB in memory and incur only write IOPS from WAL writes, bgwriter, and checkpointing.

Even a single nvme ssd can greatly exceed the maximum IOPS available to RDS, so you have to be very careful migrating database workloads to RDS.

Once your workload exceeds available throughout, latency will be terrible.

These are all hard lessons we've learned on our own. Support will not help you if you have terrible performance issues with RDS. They love to tell you to try optimizing your queries. AWS could afford to staff multiple dedicated support positions for our account, but they pocket the money instead and give terrible canned responses. If you have a tiny account, they definitely won't help you. Some people say support is great, but I've never had a good experience.

Re: AWS RDS Postmortem: Is AWS Collapsing Under its Own Weight ?

#19

Post by synicalx »

ruben81adelaide wrote: post #15
viraptor wrote: post #12

Earlier quoted context omitted.

So what was it? My money would be on (in order): Wal, logs, unexpected indices growth.

The things started to go from bad to worse when work_mem parameter has not been set up correctly. Some queries that requires a large amount of memory to process started to use disk. Once the autoscaling kicked in, even if we would have realized about it, wouldn't have helped as you are locked out of the system. The auto-scaling event was triggered earlier than our alerts for low disk available. What I don't know yet,…

Interesting problem to come across, sounds like this is a scenario where RDS and the lack of host access/visibility was a bit of a handicap. Glad you found the issue though, hopefully things go smooth next time!
Post reply on HN