Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we moved from AWS RDS to Postgres in Kubernetes

nhost.io

Post reply on HN
Print view View original

141–150 of 153 posts

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#141

Post by qeternity »

Nextgrid wrote: post #93
qeternity wrote: post #49

Earlier quoted context omitted.

> It's unlikely running it on K8S (which is itself going to run on underpowered VMs with networked storage) is going to help. What?? We run replicated Patroni on local NVMEs and it's incredibly fast.

If you have a K8S cluster running on bare-metal with directly attached disks, sure, it'll work great. I'm just pointing out that K8S by itself will not give you any performance boost if the underlying K8S nodes are the same, slow VMs with terrible IO performance.

Sure, of course. But that's a far cry from saying "on K8S (which is itself going to run on underpowered VMs with networked storage)"

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#142

Post by singron »

raffraffraff wrote: post #128
singron wrote: post #25

Earlier quoted context omitted.

RDS tops out at about 18000 IOPS since it uses a single ebs volume. Any decent ssd will do much better. E.g. a 970 evo will easily do >100K IOPS and can do more like 400K in ideal conditions. You can get that many IOPS with aurora, but the cost is exorbitant.

I believe RDS automatically stripes EBS volumes under the hood, but don't expose that information to you unless you enable enhanced metrics (it's shown under "Physical Device I/O", where you can infer the number of volumes in the stripe). I have no idea when the striping kicks in - presumable some specific volume size for gp2, or some provisioned IOPS setting. According to the link below, provisioned IOPS tops out at…

Provisioned IOPS is insanely expensive. Running 256K IOPS is $25K/month. You could outright buy multiple 970 EVOs every day and spend less money.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#143

Post by akrymski »

nunopato wrote: post #76

(Nhost) Sorry for not answering everyone individually, but I see some confusion duo to the lack of context about what we do as a company. First things first, Nhost falls into the category of backend-as-a-service. We provision and operate infrastructure at scale, and we also provide and run the necessary services for features such as user authentication and file storage, for users creating applications and businesses.…

Indeed RDS was never designed to be "re-sold", and assuming that a single PG instance will handle lots of different users is naive. Turns out if you're aiming to be an infra provider, building your own infra is the way to go. Who would have thought?

If I was launching a BaaS I wouldn't touch AWS. Grab a few Hetzner bare metal servers and setup your infra. You're leaving a massive profit margin to AWS when you don't have to.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#144

Post by dilyevsky »

jiggawatts wrote: post #99
kazen44 wrote: post #61

Earlier quoted context omitted.

> Few developers think about the cost of using the network. Developers do not seem to realise how slow the network is compared to everything else. Sure, 100gbit network itnerfaces do exist, but most servers are attached with 10gbit interfaces, and most of the actual implementations will not actually manage to hit something like 10gbit/s because of latency and window scaling. You cannot escape latency (without inventi…

It wouldn't matter if the links were 10,000 terabits! Because of the way TCP works, it has a bounded speed for small chatty transactions that is determined primarily by the latency , not the throughput. If you look at a network throughput graph from a packet capture, it looks like a sawtooth pattern. This is called slow start , and its a key feature of TCP and all similar protocols. So if a server A wants to talk to…

No one is forcing anyone to use slowstart. You can also disable things like Nagles to improve latency on small packets.

> Even on an infinite bandwidth link, sending a small blob of JSON -- say 200 kilobytes -- will take pretty much the same time as it would on a 1 Gbps link!

Technically that’s incorrect- that will take rtt+200kb/rate assuming your window is over 200kb. So depending on how large rtt is bw component may or may not be significant

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#145

Post by StreamBright »

Nextgrid wrote: post #7
radimm wrote: post #4

Having recently heard a lot of about PostgreSQL in Kubernetes (cloudNativePG for example) it always makes me wonder about the actual load and the complexity of the cluster in the question. > This is the reason why we were able to easily cope with 2M+ requests in less than 24h when Midnight Society launched This gives the answer, while it's probably not evenly distributed gives 23 req/sec (guess peak 60 - 100 might be…

> 23 req/sec (guess peak 60 - 100 might be already stretching it) That kind of load is something a decent developer laptop with an NVME drive can serve, nothing to write home about. It is sad that the "cloud" and all these supposedly "modern" DevOps systems managed to redefine the concept of "performance" for a large chunk of the industry.

I see this problem a bit more nuanced. Why does everybody starting with the assumption that the solution is SQL? You can get very far with a k:v store like S3 for example. On the top of that, if you really need SQL you can use a lot of different systems (without k8s).

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#146

Post by StreamBright »

singron wrote: post #25
Nextgrid wrote: post #7

Earlier quoted context omitted.

> 23 req/sec (guess peak 60 - 100 might be already stretching it) That kind of load is something a decent developer laptop with an NVME drive can serve, nothing to write home about. It is sad that the "cloud" and all these supposedly "modern" DevOps systems managed to redefine the concept of "performance" for a large chunk of the industry.

RDS tops out at about 18000 IOPS since it uses a single ebs volume. Any decent ssd will do much better. E.g. a 970 evo will easily do >100K IOPS and can do more like 400K in ideal conditions. You can get that many IOPS with aurora, but the cost is exorbitant.

And this is very rarely the only dimension we chose technology by.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#147

Post by dns_snek »

osigurdson wrote: post #139
dns_snek wrote: post #137

Earlier quoted context omitted.

Nitpick: 1 billion, not 1 million ;)

THz?

From the original comment, 1 GHz = 1 billion Hz = 1 billion cycles

But reading it again, I somehow missed the "In 1 ms" part, which makes it correct.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#148

Post by fmajid »

SomaticPirate wrote: post #111
fmajid wrote: post #96

Earlier quoted context omitted.

Are you using a Kubernetes PostgreSQL operator like pgo or CloudNativePG? https://proopensource.it/blog/postgresql-on-k8s-experiences

Also would like to know this. This post is a bit light on content. It sounds like they just moved to K8s from RDS. In my experience, Postgres works decently but there are sharp edges running it containerized (OOMS in subprocesses might not be caught by the container runtime, shared memory is pitifully low in docker at 64 MB by default)

From other comments, it looks like they rolled their own solution. Perhaps they had unique requirements, but it seems short-sighted to forego the automation an operator brings.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#149

Post by xwowsersx »

CodesInChaos wrote: post #140
xwowsersx wrote: post #21

Earlier quoted context omitted.

Makes sense, thank you. Sounds like M class is the way to go as other commenter suggested. Also, yes. There are many awful queries that I'm aware of and working to correct.

I'd look into Memory Optimized (R*) instances.

Thank you. I think this makes a lot more sense actually. I can go to db.r6g.xlarge and double the memory to 32gb from the t3.xlarge I'm currently on for an additional ~$117/month vs getting to 32gb with the db.m5.2xlarge for $308 more per month. Also, looks like X2g among the memory optimized is the lowest price per GiB of RAM (for MySQL, MariaDB, and PostgreSQL). Thoughts on the X2g? The db.x2g.large, for example, doubles memory AND network performance (Gbps) vs current DB for less than $30 more per month. Does drop vCPU down to 2 from 4, which might not matter given where CPU utilization seems to spike to (~50% at peak) https://ibb.co/WsxN6D3

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#150

Post by MuffinFlavored »

ahachete wrote: post #125
MuffinFlavored wrote: post #112

Earlier quoted context omitted.

> Having recently heard a lot of about PostgreSQL in Kubernetes I could never get a straight answer on whether running a database in a container (and mounting the storage volume through a bind mount/network drive or whatever) came with a performance hit compared to running it as a systemd service for example.

In case you are interested, I blogged about it last year: https://thenewstack.io/kubernetes-will-revolutionize-enterpr... TL;DR performance impact should be negligible, could be even slightly negative compared to a VM (when running K8s on bare metal).

What's the best way to simply mount the storage volume needed for Postgres to be performant?
Post reply on HN