11–20 of 153 posts
Ah, the 'ol sunk cost fallacy of infrastructure. We are already investing in supporting K8s, so let's throw the databases in there too. Couldn't possibly be that much work. Sure, a decade-old dedicated team at a billion-dollar multinational corporation has honed a solution designed to support hundreds of thousands of customers with high availability, and we could pay a little bit extra money to spin up a new database…
I've recently been spending a fair amount of time trying to improve query performance on RDS. This includes reviewing and optimizing particularly nasty queries, tuning PG configuration (min_wal_size, random_page_cost, work_mem, etc). I am using a db.t3.xlarge with general purpose SSD (gp2) for a web server that sees moderate writes and a lot of reads. I know there's no real way to know other than through testing, but…
I'd recommend hopping off of t3 asap if you're searching for performance gains - performance can be extremely variable (by design). M class will even you out. General storage IOPS is governed by your provisioned storage size. You can again get much more consistent performance by using provisioned IOPS. Feel free to email me if you want to chat through things specific to your env - email is in my about:
I've recently been spending a fair amount of time trying to improve query performance on RDS. This includes reviewing and optimizing particularly nasty queries, tuning PG configuration (min_wal_size, random_page_cost, work_mem, etc). I am using a db.t3.xlarge with general purpose SSD (gp2) for a web server that sees moderate writes and a lot of reads. I know there's no real way to know other than through testing, but…
It's hard to say without metrics; what does your CPU load look like? In general, unless your CPU is often maxing out, changing the CPU is unlikely to help, so you're left with either memory or IO. Unused memory on Linux will be automatically used to cache IO operations, and you can also tweak PG itself to use more memory during queries (search for "work_mem", though there are others). If your workload is read-heavy,…
So really we don’t know how much RDS was a problem compared to the the tenant distribution.
For the purposes of an article like this it would be nice if the two steps were separate or they had synthetic benchmarks of the various options.
But I understand why they just moved forward. They said they consulted experts, it would also be nice to discuss some of what they looked or asked about.
Ah, the 'ol sunk cost fallacy of infrastructure. We are already investing in supporting K8s, so let's throw the databases in there too. Couldn't possibly be that much work. Sure, a decade-old dedicated team at a billion-dollar multinational corporation has honed a solution designed to support hundreds of thousands of customers with high availability, and we could pay a little bit extra money to spin up a new database…
And when it all goes bottoms up it will be much more difficult to resolve.
Probably there is also an 80/20 for most users where it's not awful if you can restore from a cold storage, say 12h, backup.
Earlier quoted context omitted.
It's hard to say without metrics; what does your CPU load look like? In general, unless your CPU is often maxing out, changing the CPU is unlikely to help, so you're left with either memory or IO. Unused memory on Linux will be automatically used to cache IO operations, and you can also tweak PG itself to use more memory during queries (search for "work_mem", though there are others). If your workload is read-heavy,…
Thanks very much for the reply. CPU is not often maxing out. Here's a graph of max CPU utilization from the last week https://ibb.co/tzw5p3L
I suggest upgrading to an instance type which gives you 32GB or more of memory. You'll get a bigger CPU along with it as well, but don't make the CPU your priority, it's not your main bottleneck at the moment.
$0.50 per extra GB seems high, especially for a storage-intensive app. Given the cost of cloud Object Storage services it doesn't seem to make much sense. Examples of alternatives for managed Postgres: * Supabase is $0.125 per GB * DigitalOcean managed Postgres is ~$0.35 per GB
Ah, the 'ol sunk cost fallacy of infrastructure. We are already investing in supporting K8s, so let's throw the databases in there too. Couldn't possibly be that much work. Sure, a decade-old dedicated team at a billion-dollar multinational corporation has honed a solution designed to support hundreds of thousands of customers with high availability, and we could pay a little bit extra money to spin up a new database…
Source: AWS user on very very large scale stuff for about 10 years now. It's not magic or perfection. It's just someone else's pile of problems that are lurking. The only consolation is they appear to try slightly harder than the datacentres that we replaced.