Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we moved from AWS RDS to Postgres in Kubernetes

nhost.io

Post reply on HN
Print view View original

21–30 of 153 posts

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#21

Post by xwowsersx »

Nextgrid wrote: post #17
xwowsersx wrote: post #13

Earlier quoted context omitted.

Thanks very much for the reply. CPU is not often maxing out. Here's a graph of max CPU utilization from the last week https://ibb.co/tzw5p3L

You've got some spikes that could signify some large or unoptimized queries, but otherwise yes, the CPU doesn't look that hot. I suggest upgrading to an instance type which gives you 32GB or more of memory. You'll get a bigger CPU along with it as well, but don't make the CPU your priority, it's not your main bottleneck at the moment.

Makes sense, thank you. Sounds like M class is the way to go as other commenter suggested. Also, yes. There are many awful queries that I'm aware of and working to correct.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#23

Post by xani_ »

0xbadcafebee wrote: post #6

Ah, the 'ol sunk cost fallacy of infrastructure. We are already investing in supporting K8s, so let's throw the databases in there too. Couldn't possibly be that much work. Sure, a decade-old dedicated team at a billion-dollar multinational corporation has honed a solution designed to support hundreds of thousands of customers with high availability, and we could pay a little bit extra money to spin up a new database…

No post body was provided.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#24

Post by xani_ »

radimm wrote: post #4

Having recently heard a lot of about PostgreSQL in Kubernetes (cloudNativePG for example) it always makes me wonder about the actual load and the complexity of the cluster in the question. > This is the reason why we were able to easily cope with 2M+ requests in less than 24h when Midnight Society launched This gives the answer, while it's probably not evenly distributed gives 23 req/sec (guess peak 60 - 100 might be…

It's essentially just a process running in a cgroup so performance shouldn't be all that different than bare metal/VM postgresql.

Main difference would be storage speed and how it exactly is attached to a container.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#25

Post by singron »

Nextgrid wrote: post #7
radimm wrote: post #4

Having recently heard a lot of about PostgreSQL in Kubernetes (cloudNativePG for example) it always makes me wonder about the actual load and the complexity of the cluster in the question. > This is the reason why we were able to easily cope with 2M+ requests in less than 24h when Midnight Society launched This gives the answer, while it's probably not evenly distributed gives 23 req/sec (guess peak 60 - 100 might be…

> 23 req/sec (guess peak 60 - 100 might be already stretching it) That kind of load is something a decent developer laptop with an NVME drive can serve, nothing to write home about. It is sad that the "cloud" and all these supposedly "modern" DevOps systems managed to redefine the concept of "performance" for a large chunk of the industry.

RDS tops out at about 18000 IOPS since it uses a single ebs volume. Any decent ssd will do much better. E.g. a 970 evo will easily do >100K IOPS and can do more like 400K in ideal conditions.

You can get that many IOPS with aurora, but the cost is exorbitant.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#26

Post by suggala »

0xbadcafebee wrote: post #6

Ah, the 'ol sunk cost fallacy of infrastructure. We are already investing in supporting K8s, so let's throw the databases in there too. Couldn't possibly be that much work. Sure, a decade-old dedicated team at a billion-dollar multinational corporation has honed a solution designed to support hundreds of thousands of customers with high availability, and we could pay a little bit extra money to spin up a new database…

AWS RDS is 10x slower than BareMetal MySQL (both reads and writes). Slowness is mainly due to the reason that Storage is over network for RDS.

Not bad to invest some extra time to get better performance.

You are falling to “Appeal to antiquity” fallacy if you think something old is better.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#27

Post by wbl »

xani_ wrote: post #23
0xbadcafebee wrote: post #6

Ah, the 'ol sunk cost fallacy of infrastructure. We are already investing in supporting K8s, so let's throw the databases in there too. Couldn't possibly be that much work. Sure, a decade-old dedicated team at a billion-dollar multinational corporation has honed a solution designed to support hundreds of thousands of customers with high availability, and we could pay a little bit extra money to spin up a new database…

Quoted post unavailable.

Running a statefull service in K8S is its own ball of wax

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#28

Post by mhuffman »

Nextgrid wrote: post #7
radimm wrote: post #4

Having recently heard a lot of about PostgreSQL in Kubernetes (cloudNativePG for example) it always makes me wonder about the actual load and the complexity of the cluster in the question. > This is the reason why we were able to easily cope with 2M+ requests in less than 24h when Midnight Society launched This gives the answer, while it's probably not evenly distributed gives 23 req/sec (guess peak 60 - 100 might be…

> 23 req/sec (guess peak 60 - 100 might be already stretching it) That kind of load is something a decent developer laptop with an NVME drive can serve, nothing to write home about. It is sad that the "cloud" and all these supposedly "modern" DevOps systems managed to redefine the concept of "performance" for a large chunk of the industry.

It does depend on the architecture and framework they are using imo. I have a single Hetzer machine with spinning plate HDs that serves between 1-2 million requests per day hitting DB and ML models and rarely every gets over 1% CPU usage. I have pressure-tested it to around 3k reqs/sec. On the other hand I have seen WP and CodeIgniter setups that even with 5 copies running on the largest AWS instances available, "optimized" to the hilt, caching everywhere possible, etc. absolute crumble under the load of 3k req per min. (not sec ... min).

Many frameworks that make early development easy fuck you later during growth with ORM calls, tons of unnecessary text in the DB, etc.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#29

Post by peterangular »

xani_ wrote: post #23
0xbadcafebee wrote: post #6

Ah, the 'ol sunk cost fallacy of infrastructure. We are already investing in supporting K8s, so let's throw the databases in there too. Couldn't possibly be that much work. Sure, a decade-old dedicated team at a billion-dollar multinational corporation has honed a solution designed to support hundreds of thousands of customers with high availability, and we could pay a little bit extra money to spin up a new database…

Quoted post unavailable.

I'd posit that it's not as simple. Maybe if you're just cranking out your one-off app or something of the sort...

But getting a good replication setup that's HA, potentially across multiple regions/zones, all abstracted under K8s - yea. That's not trivial. And, it can go very wrong.

> I bet you also hate on people making their own espresso instead of just going to starbucks

This is just unnecessary.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#30

Post by brand »

radimm wrote: post #4

Having recently heard a lot of about PostgreSQL in Kubernetes (cloudNativePG for example) it always makes me wonder about the actual load and the complexity of the cluster in the question. > This is the reason why we were able to easily cope with 2M+ requests in less than 24h when Midnight Society launched This gives the answer, while it's probably not evenly distributed gives 23 req/sec (guess peak 60 - 100 might be…

I’ve personally deployed O(TBs) and O(10^4 TPS) Postgres clusters on Kubernetes with a CNPG-style operator based deployment. There are some subtleties to it but it’s not exceeding complicated, and a good project like CNPG goes a long way to shaving off those sharp edges. As other commenters have suggested it’s good to really understand Kubernetes if you want to do it, though.
Post reply on HN