Why we moved from AWS RDS to Postgres in Kubernetes
1–10 of 153 posts
Re: Why we moved from AWS RDS to Postgres in Kubernetes
#2I've recently been spending a fair amount of time trying to improve query performance on RDS. This includes reviewing and optimizing particularly nasty queries, tuning PG configuration (min_wal_size, random_page_cost, work_mem, etc). I am using a db.t3.xlarge with general purpose SSD (gp2) for a web server that sees moderate writes and a lot of reads. I know there's no real way to know other than through testing, but I'm not clear on which instance type best serves our needs — I think it may very well be the case that the t3 family isn't fit for our purposes. I'm also unclear on whether we ought to switch to provisioned IOPS SSD. Does anyone have any general pointers here? I know the question is pretty open-ended, but would be great if anyone has general advice from personal experience?
Re: Why we moved from AWS RDS to Postgres in Kubernetes
#3I've recently been spending a fair amount of time trying to improve query performance on RDS. This includes reviewing and optimizing particularly nasty queries, tuning PG configuration (min_wal_size, random_page_cost, work_mem, etc). I am using a db.t3.xlarge with general purpose SSD (gp2) for a web server that sees moderate writes and a lot of reads. I know there's no real way to know other than through testing, but…
I'd recommend hopping off of t3 asap if you're searching for performance gains - performance can be extremely variable (by design). M class will even you out.
General storage IOPS is governed by your provisioned storage size. You can again get much more consistent performance by using provisioned IOPS.
Feel free to email me if you want to chat through things specific to your env - email is in my about:
Re: Why we moved from AWS RDS to Postgres in Kubernetes
#4Having recently heard a lot of about PostgreSQL in Kubernetes (cloudNativePG for example) it always makes me wonder about the actual load and the complexity of the cluster in the question.
> This is the reason why we were able to easily cope with 2M+ requests in less than 24h when Midnight Society launched
This gives the answer, while it's probably not evenly distributed gives 23 req/sec (guess peak 60 - 100 might be already stretching it). Always wonder about use cases around 3 - 5k req/sec as minimum.
[edit] PS: not really ditching neither k8s pg nor AWS RDS or similar solutions. Just being curious.
Re: Why we moved from AWS RDS to Postgres in Kubernetes
#5I've recently been spending a fair amount of time trying to improve query performance on RDS. This includes reviewing and optimizing particularly nasty queries, tuning PG configuration (min_wal_size, random_page_cost, work_mem, etc). I am using a db.t3.xlarge with general purpose SSD (gp2) for a web server that sees moderate writes and a lot of reads. I know there's no real way to know other than through testing, but…
It's hard to say without metrics; what does your CPU load look like? In general, unless your CPU is often maxing out, changing the CPU is unlikely to help, so you're left with either memory or IO.
Unused memory on Linux will be automatically used to cache IO operations, and you can also tweak PG itself to use more memory during queries (search for "work_mem", though there are others).
If your workload is read-heavy, just giving it more memory so that the majority of your dataset is always in the kernel IO cache will give you an immediate performance boost, without even having to tweak PG's config (though that might help even further). This won't transfer to writes - those still require an actual, uncached IO operation to complete (unless you want to put your data at risk, in which case there are parameters that can be used to override that).
For write-heavy workloads, you will need to upgrade IO; there's no way around the "provisioned IOPS" disks.
Re: Why we moved from AWS RDS to Postgres in Kubernetes
#6Ah, the 'ol sunk cost fallacy of infrastructure. We are already investing in supporting K8s, so let's throw the databases in there too. Couldn't possibly be that much work.
Sure, a decade-old dedicated team at a billion-dollar multinational corporation has honed a solution designed to support hundreds of thousands of customers with high availability, and we could pay a little bit extra money to spin up a new database per tenant that's a little bit less flexible, ..... or we could reinvent everything they do on our own software platform and expect the same results. All it'll cost us is extra expertise, extra staff, extra time, extra money, extra planning, and extra operations. But surely it will improve our product dramatically.
Re: Why we moved from AWS RDS to Postgres in Kubernetes
#7Having recently heard a lot of about PostgreSQL in Kubernetes (cloudNativePG for example) it always makes me wonder about the actual load and the complexity of the cluster in the question. > This is the reason why we were able to easily cope with 2M+ requests in less than 24h when Midnight Society launched This gives the answer, while it's probably not evenly distributed gives 23 req/sec (guess peak 60 - 100 might be…
> 23 req/sec (guess peak 60 - 100 might be already stretching it)
That kind of load is something a decent developer laptop with an NVME drive can serve, nothing to write home about.
It is sad that the "cloud" and all these supposedly "modern" DevOps systems managed to redefine the concept of "performance" for a large chunk of the industry.
Re: Why we moved from AWS RDS to Postgres in Kubernetes
#8I would love to see the monitoring on this.
Network IOPs and NAT nastiness or disk IO the bigger issue ?
Re: Why we moved from AWS RDS to Postgres in Kubernetes
#9Having recently heard a lot of about PostgreSQL in Kubernetes (cloudNativePG for example) it always makes me wonder about the actual load and the complexity of the cluster in the question. > This is the reason why we were able to easily cope with 2M+ requests in less than 24h when Midnight Society launched This gives the answer, while it's probably not evenly distributed gives 23 req/sec (guess peak 60 - 100 might be…
> 23 req/sec (guess peak 60 - 100 might be already stretching it) That kind of load is something a decent developer laptop with an NVME drive can serve, nothing to write home about. It is sad that the "cloud" and all these supposedly "modern" DevOps systems managed to redefine the concept of "performance" for a large chunk of the industry.
It depends a lot on the backend architecture. Number of DB requests per web request can also be high due to the pathological cases in some ORMs which can result in N+1 query problems or eagerly fetching entire object hierarchies. Such problems in application code can get brushed under the carpet due to "magical" autoscaling (be it RDS or K8s). There can also be fanout to async services/job queues which will in turn run even more DB queries.
Re: Why we moved from AWS RDS to Postgres in Kubernetes
#10$0.50 per extra GB seems high, especially for a storage-intensive app. Given the cost of cloud Object Storage services it doesn't seem to make much sense.
Examples of alternatives for managed Postgres:
* Supabase is $0.125 per GB
* DigitalOcean managed Postgres is ~$0.35 per GB