Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we moved from AWS RDS to Postgres in Kubernetes

nhost.io

Post reply on HN
Print view View original

121–130 of 153 posts

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#121

Post by raffraffraff »

robertlagrant wrote: post #72
rrampage wrote: post #9

Earlier quoted context omitted.

It depends a lot on the backend architecture. Number of DB requests per web request can also be high due to the pathological cases in some ORMs which can result in N+1 query problems or eagerly fetching entire object hierarchies. Such problems in application code can get brushed under the carpet due to "magical" autoscaling (be it RDS or K8s). There can also be fanout to async services/job queues which will in turn r…

I don't think K8s at least will autoscale quickly enough to mask something like that.

Autoscaling is slow. If you're using AWS autoscaling group, decisions are based on several different metrics that are typically averaged over a period. If the instance pool size is increased, that fact gets picked up by yet another event loop that runs periodically, and actually starts instances. So there are multiple chained delays before the instance is actually launched. In practice, even if your instances have extremely fast start-up and can begin processing the queue quickly, the job in the queue could be waiting 4+ minutes to get picked up, in a scale-to-zero situation. You've also got things like cooldown periods to ensure that you are not flapping.

With k8s you have more control over knobs and switches, and you don't have an instance start-up delay, but the same type of metrics and event loops are used, particularly if you're using an external metric (eg SQS queue depth) in your calculations.

Some type of predictive and/or scheduled scaling can reduce delays at the expense of potentially higher cost.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#122

Post by raffraffraff »

MBCook wrote: post #14

So they switch from one giant RDS instance with all tenants per AZ to per-tenant PG in Kubernetes. So really we don’t know how much RDS was a problem compared to the the tenant distribution. For the purposes of an article like this it would be nice if the two steps were separate or they had synthetic benchmarks of the various options. But I understand why they just moved forward. They said they consulted experts, it…

Yeah. I mean, if you're going to use AWS database service for this use case, something that automatically scales based on load makes more sense, like Aurora Serverless. But that's also expensive. Regardless of cost, plain RDS isn't the right solution here as all.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#123

Post by movedx »

xyzzy_plugh wrote: post #118

If the cost of operating a postgres database is eating into your margins so much (and you can't simply adjust your prices to eat the difference) then I would suspect the wrong technology is in place. Sure, RDS is expensive, but it's also quite well done. Almost every cloud platform service is more expensive than doing it yourself. No surprise here. In the past I've deployed SQLite over Postgres for cost cutting reaso…

> Almost every cloud platform service is more expensive than doing it yourself. No surprise here.

In a business environment, this is actually not true unless you consider the extreme long term.

A Multi-AZ MySQL RDS instance of size db.m1.large (2x vCPU, 7.5GB of RAM), a 500GB standard disk, and an on-demand pricing model with 100% monthly utilization, will cost you approx. US$7,000 per year (rounding up.) That price gets you almost everything you can imagine from that service.

US$7,000 wouldn't get you my services for the time needed to setup a service that came even 30% as close in terms of reliability, feature parity and support.

RDS is not expensive (in the right environment.)

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#124

Post by ahachete »

osigurdson wrote: post #107
kazen44 wrote: post #61

Earlier quoted context omitted.

> Few developers think about the cost of using the network. Developers do not seem to realise how slow the network is compared to everything else. Sure, 100gbit network itnerfaces do exist, but most servers are attached with 10gbit interfaces, and most of the actual implementations will not actually manage to hit something like 10gbit/s because of latency and window scaling. You cannot escape latency (without inventi…

Light travels 300km in 1 millisecond. Intra datacenter latency is not bounded by physics. It is bounded by current technology.

Light travels much slower (~1.5x slower) on a fiber optic, due to the refractive index (~ 1.5) of the fiber.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#125

Post by ahachete »

MuffinFlavored wrote: post #112
radimm wrote: post #4

Having recently heard a lot of about PostgreSQL in Kubernetes (cloudNativePG for example) it always makes me wonder about the actual load and the complexity of the cluster in the question. > This is the reason why we were able to easily cope with 2M+ requests in less than 24h when Midnight Society launched This gives the answer, while it's probably not evenly distributed gives 23 req/sec (guess peak 60 - 100 might be…

> Having recently heard a lot of about PostgreSQL in Kubernetes I could never get a straight answer on whether running a database in a container (and mounting the storage volume through a bind mount/network drive or whatever) came with a performance hit compared to running it as a systemd service for example.

In case you are interested, I blogged about it last year: https://thenewstack.io/kubernetes-will-revolutionize-enterpr...

TL;DR performance impact should be negligible, could be even slightly negative compared to a VM (when running K8s on bare metal).

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#126

Post by eptcyka »

raffraffraff wrote: post #122
MBCook wrote: post #14

So they switch from one giant RDS instance with all tenants per AZ to per-tenant PG in Kubernetes. So really we don’t know how much RDS was a problem compared to the the tenant distribution. For the purposes of an article like this it would be nice if the two steps were separate or they had synthetic benchmarks of the various options. But I understand why they just moved forward. They said they consulted experts, it…

Yeah. I mean, if you're going to use AWS database service for this use case, something that automatically scales based on load makes more sense, like Aurora Serverless. But that's also expensive. Regardless of cost, plain RDS isn't the right solution here as all.

Yes, that's basically the whole point of the article - an assumption they may not have made all too consciously to use RDS turned into a bad decision they sought to rectify.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#127

Post by eptcyka »

Nextgrid wrote: post #7
radimm wrote: post #4

Having recently heard a lot of about PostgreSQL in Kubernetes (cloudNativePG for example) it always makes me wonder about the actual load and the complexity of the cluster in the question. > This is the reason why we were able to easily cope with 2M+ requests in less than 24h when Midnight Society launched This gives the answer, while it's probably not evenly distributed gives 23 req/sec (guess peak 60 - 100 might be…

> 23 req/sec (guess peak 60 - 100 might be already stretching it) That kind of load is something a decent developer laptop with an NVME drive can serve, nothing to write home about. It is sad that the "cloud" and all these supposedly "modern" DevOps systems managed to redefine the concept of "performance" for a large chunk of the industry.

NVME? You can serve this from a raspberry pi.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#128

Post by raffraffraff »

singron wrote: post #25
Nextgrid wrote: post #7

Earlier quoted context omitted.

> 23 req/sec (guess peak 60 - 100 might be already stretching it) That kind of load is something a decent developer laptop with an NVME drive can serve, nothing to write home about. It is sad that the "cloud" and all these supposedly "modern" DevOps systems managed to redefine the concept of "performance" for a large chunk of the industry.

RDS tops out at about 18000 IOPS since it uses a single ebs volume. Any decent ssd will do much better. E.g. a 970 evo will easily do >100K IOPS and can do more like 400K in ideal conditions. You can get that many IOPS with aurora, but the cost is exorbitant.

I believe RDS automatically stripes EBS volumes under the hood, but don't expose that information to you unless you enable enhanced metrics (it's shown under "Physical Device I/O", where you can infer the number of volumes in the stripe). I have no idea when the striping kicks in - presumable some specific volume size for gp2, or some provisioned IOPS setting.

According to the link below, provisioned IOPS tops out at 256000:

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_....

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#129

Post by vbezhenar »

SOLAR_FIELDS wrote: post #85
cloudbee wrote: post #83

Earlier quoted context omitted.

And what are your cost savings from RDS perspective. I'd a similar problem where we'd to provision like 5 databases for 5 different teams. RDS is really expensive. And your solution is open source ? I would like to try.

RDS and similar managed databases are over half of our total cloud bill at my place of work. Managed databases in general are really expensive .

Is there any particular reason for managed databases being expensive or they just charge because they can?

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#130

Post by KronisLV »

api wrote: post #58
qeternity wrote: post #52

These threads are always full of people who have always used an AWS/GCP/Azure service, or have never actually run the service themselves. Running HA Postgres is not easy...but at any sort of scale where this stuff matters, nothing is easy. It's not as if AWS has 100% uptime, nor is it super cheap/performant. There are tradeoffs for everyone's use-case but every thread is full of people at one end of the cloud / roll-…

I wonder how many people use things like CockroachDB, Yugabyte, or TiDB? They're at least in theory far easier to run in HA configurations at the cost of some additional overhead and in some cases more limited SQL functionality. They seem like a huge step up from the arcane "1980s Unix" nightmare of Postgres clustering but I don't hear about them that much. Are they not used much or are their users just happy and qui…

> I wonder how many people use things like CockroachDB, Yugabyte, or TiDB?

TiDB is a pretty interesting project, but there are a few limitations that should be taken into account when trying to use it: https://docs.pingcap.com/tidb/stable/mysql-compatibility

A lot of these are tradeoffs that will affect how a database can be architected, such as having no access to foreign keys and thus needing to think about any sort of consistency and not leaving orphaned data at the application level.

Post reply on HN