Having recently heard a lot of about PostgreSQL in Kubernetes (cloudNativePG for example) it always makes me wonder about the actual load and the complexity of the cluster in the question. > This is the reason why we were able to easily cope with 2M+ requests in less than 24h when Midnight Society launched This gives the answer, while it's probably not evenly distributed gives 23 req/sec (guess peak 60 - 100 might be…
> 23 req/sec (guess peak 60 - 100 might be already stretching it) That kind of load is something a decent developer laptop with an NVME drive can serve, nothing to write home about. It is sad that the "cloud" and all these supposedly "modern" DevOps systems managed to redefine the concept of "performance" for a large chunk of the industry.
Why we moved from AWS RDS to Postgres in Kubernetes
91–100 of 153 posts
Re: Why we moved from AWS RDS to Postgres in Kubernetes
#92I've recently been spending a fair amount of time trying to improve query performance on RDS. This includes reviewing and optimizing particularly nasty queries, tuning PG configuration (min_wal_size, random_page_cost, work_mem, etc). I am using a db.t3.xlarge with general purpose SSD (gp2) for a web server that sees moderate writes and a lot of reads. I know there's no real way to know other than through testing, but…
Of course if you need to recover quickly in a disaster you'll want a hot standby or replica. Still may be cheaper than PIOPs. (Especially if you need HA anyway.)
Re: Why we moved from AWS RDS to Postgres in Kubernetes
#93Earlier quoted context omitted.
It's unlikely running it on K8S (which is itself going to run on underpowered VMs with networked storage) is going to help. If you're gonna spend effort in running Postgres manually, do it on bare-metal and at least get some reward out of it (performance and reduced cost).
> It's unlikely running it on K8S (which is itself going to run on underpowered VMs with networked storage) is going to help. What?? We run replicated Patroni on local NVMEs and it's incredibly fast.
Re: Why we moved from AWS RDS to Postgres in Kubernetes
#94$0.50 per extra GB seems high, especially for a storage-intensive app. Given the cost of cloud Object Storage services it doesn't seem to make much sense. Examples of alternatives for managed Postgres: * Supabase is $0.125 per GB * DigitalOcean managed Postgres is ~$0.35 per GB
For small startups, my DR/HA plan is hourly delta snapshots of the whole volume.
GCP, AWS and Azure all make this possible.
Re: Why we moved from AWS RDS to Postgres in Kubernetes
#95These threads are always full of people who have always used an AWS/GCP/Azure service, or have never actually run the service themselves. Running HA Postgres is not easy...but at any sort of scale where this stuff matters, nothing is easy. It's not as if AWS has 100% uptime, nor is it super cheap/performant. There are tradeoffs for everyone's use-case but every thread is full of people at one end of the cloud / roll-…
Honestly, that's what I initially thought trying to run ha postgres on k8s, but zalando's postgres operator made things so much easier (maybe even easier than RDS). Very easy to rollout as many postgres clusters with whatever size you want. We've been running our production db on it for the last 6 months or so, no outage yet. Though I guess if you have to have a very custom setup, it might be more difficult.
Re: Why we moved from AWS RDS to Postgres in Kubernetes
#96(Nhost) Sorry for not answering everyone individually, but I see some confusion duo to the lack of context about what we do as a company. First things first, Nhost falls into the category of backend-as-a-service. We provision and operate infrastructure at scale, and we also provide and run the necessary services for features such as user authentication and file storage, for users creating applications and businesses.…
Re: Why we moved from AWS RDS to Postgres in Kubernetes
#97(Nhost) Sorry for not answering everyone individually, but I see some confusion duo to the lack of context about what we do as a company. First things first, Nhost falls into the category of backend-as-a-service. We provision and operate infrastructure at scale, and we also provide and run the necessary services for features such as user authentication and file storage, for users creating applications and businesses.…
And what are your cost savings from RDS perspective. I'd a similar problem where we'd to provision like 5 databases for 5 different teams. RDS is really expensive. And your solution is open source ? I would like to try.
A while ago, I worked for a company that offered a hosted version of their application that required Postgres, etcd, Kubernetes, etc. It was set up so that every customer got their own GCP project, containing a K8s cluster, Cloud Storage, and a Postgres instance, The k8s cluster ("workspace") then contained dedicated nodes (4vCPU x 16G RAM at a minimum, autoscaling up according to their workload including GPU compute), SSDs, a public-facing LoadBalancer, etc. This is good for per-customer isolation, but quite costly at idle, on the order of several hundred dollars a month. Users expect this kind of isolation (but need the SOC2 and similar checkmark for sure), but they don't expect to be charged when they're not running anything, which was a problem for us.
If I was doing this again, I would do it this way, at least for the MVP. One option is to make the application multi-tenant aware, and isolate at the application level instead of at the GCP project level. This might be more difficult to get certified and might not meet everyone's HIPAA-like compliance goals, but is a good starting point, especially for free trials.
The other option that was very appealing to me is to give each user a VM that just gets de-scheduled when no requests are being made. Instead of k8s managing nodes, nodes would manage k8s. The downside there is that cluster size is limited to whatever the largest node you can buy is, but honestly, 448vCPUs is a ton (AWS's max instance size at the moment), so it's a very workable solution. When users sign up, create a VM image that runs K8s, Minio, Postgres, etc. and route traffic to it with a shared L7 router/front proxy. If their workloads autoscale up, freeze and migrate the VM to a machine with more resources. If they're not using it for a while, freeze it completely, and reprogram your front proxy to point at a program that waits for an RPC / web request and starts up the VM when one comes in. Now your idle cost is the cost of your block storage, modulo deduplication, instead of dedicated CPU cores and RAM. You also get a lot of knobs to control your actual compute cost; you aren't reliant on your users provisioning spot instances from their cloud provider, you can just tell cron jobs to run when CPU load is lowest, or set your own rate to incentive off-peak usage. And, you can pretty much get away with charging nothing for idle instances, limit free trials in aggregate to X CPU cores, etc. I think it would have been good, though complex.
TL;DR: RDS is a highly-available always-on service. But customers might not want HA or always-on. By being able to turn off the database at the right moment, you can save a lot of money on compute, which makes things like good free trials more economically viable. I think OP is on the right track to a successful k8s-based business and wish them great luck!
Re: Why we moved from AWS RDS to Postgres in Kubernetes
#98I've recently been spending a fair amount of time trying to improve query performance on RDS. This includes reviewing and optimizing particularly nasty queries, tuning PG configuration (min_wal_size, random_page_cost, work_mem, etc). I am using a db.t3.xlarge with general purpose SSD (gp2) for a web server that sees moderate writes and a lot of reads. I know there's no real way to know other than through testing, but…
I'd recommend hopping off of t3 asap if you're searching for performance gains - performance can be extremely variable (by design). M class will even you out. General storage IOPS is governed by your provisioned storage size. You can again get much more consistent performance by using provisioned IOPS. Feel free to email me if you want to chat through things specific to your env - email is in my about:
Re: Why we moved from AWS RDS to Postgres in Kubernetes
#99Earlier quoted context omitted.
I can't blame it on "cloud", though it's not helping that there are an awful lot of cloud services that claim to be "high performance" and are often mediumish at best. But in general I see a lot of ignorance in the developer community as to how fast things should be able to run, even in terms of reading local files and doing local manipulations with no "cloud" in sight. Honestly, if I had to pin it on just one thing,…
> Few developers think about the cost of using the network. Developers do not seem to realise how slow the network is compared to everything else. Sure, 100gbit network itnerfaces do exist, but most servers are attached with 10gbit interfaces, and most of the actual implementations will not actually manage to hit something like 10gbit/s because of latency and window scaling. You cannot escape latency (without inventi…
If you look at a network throughput graph from a packet capture, it looks like a sawtooth pattern. This is called slow start, and its a key feature of TCP and all similar protocols.
So if a server A wants to talk to a server B, it sends 8 packets, waits for a response, then sends 16 packets, waits, 24 packets, waits, and so on until a response is dropped. It then resets to 8 packets. There are lots of variations on this algorithm, such as using a "cubic" curve instead of a linear curve, but the end result is pretty much the same.
Even on an infinite bandwidth link, sending a small blob of JSON -- say 200 kilobytes -- will take pretty much the same time as it would on a 1 Gbps link!
As a side effect of this, anything that reduces latency can have a dramatic effect on effective bandwidth. I've seen some applications triple in speed simply because I enabled "Accelerated Networking" in Azure and used a Proximity Placement Group.
Re: Why we moved from AWS RDS to Postgres in Kubernetes
#100Do I have to manually upgrade my old instances?