Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we moved from AWS RDS to Postgres in Kubernetes

nhost.io

Post reply on HN
Print view View original

71–80 of 153 posts

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#71

Post by 988747 »

qeternity wrote: post #52

These threads are always full of people who have always used an AWS/GCP/Azure service, or have never actually run the service themselves. Running HA Postgres is not easy...but at any sort of scale where this stuff matters, nothing is easy. It's not as if AWS has 100% uptime, nor is it super cheap/performant. There are tradeoffs for everyone's use-case but every thread is full of people at one end of the cloud / roll-…

I've been successfully running Postgres in Kubernetes with the Operator from Crunchy Data. It makes HA setup really easy with a tool called Patroni, which basically takes care of all the hard stuff. Running 1 primary and 2 replicas is really no harder than running single-node Postgres.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#72

Post by robertlagrant »

rrampage wrote: post #9
Nextgrid wrote: post #7

Earlier quoted context omitted.

> 23 req/sec (guess peak 60 - 100 might be already stretching it) That kind of load is something a decent developer laptop with an NVME drive can serve, nothing to write home about. It is sad that the "cloud" and all these supposedly "modern" DevOps systems managed to redefine the concept of "performance" for a large chunk of the industry.

It depends a lot on the backend architecture. Number of DB requests per web request can also be high due to the pathological cases in some ORMs which can result in N+1 query problems or eagerly fetching entire object hierarchies. Such problems in application code can get brushed under the carpet due to "magical" autoscaling (be it RDS or K8s). There can also be fanout to async services/job queues which will in turn r…

I don't think K8s at least will autoscale quickly enough to mask something like that.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#73

Post by briffle »

kazen44 wrote: post #61
jerf wrote: post #34

Earlier quoted context omitted.

I can't blame it on "cloud", though it's not helping that there are an awful lot of cloud services that claim to be "high performance" and are often mediumish at best. But in general I see a lot of ignorance in the developer community as to how fast things should be able to run, even in terms of reading local files and doing local manipulations with no "cloud" in sight. Honestly, if I had to pin it on just one thing,…

> Few developers think about the cost of using the network. Developers do not seem to realise how slow the network is compared to everything else. Sure, 100gbit network itnerfaces do exist, but most servers are attached with 10gbit interfaces, and most of the actual implementations will not actually manage to hit something like 10gbit/s because of latency and window scaling. You cannot escape latency (without inventi…

it seems most of the tools for running postgresql in K8s seem to just default to creating a new copy of the DB at the drop of a hat. When your DB is in the multi-TB sizes, that can come with a noticable cost in network fees, plus a very long delay, even on modern fast networks.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#74

Post by radimm »

brand wrote: post #30
radimm wrote: post #4

Having recently heard a lot of about PostgreSQL in Kubernetes (cloudNativePG for example) it always makes me wonder about the actual load and the complexity of the cluster in the question. > This is the reason why we were able to easily cope with 2M+ requests in less than 24h when Midnight Society launched This gives the answer, while it's probably not evenly distributed gives 23 req/sec (guess peak 60 - 100 might be…

I’ve personally deployed O(TBs) and O(10^4 TPS) Postgres clusters on Kubernetes with a CNPG-style operator based deployment. There are some subtleties to it but it’s not exceeding complicated, and a good project like CNPG goes a long way to shaving off those sharp edges. As other commenters have suggested it’s good to really understand Kubernetes if you want to do it, though.

Thanks for the confirmation. As mentioned I'm not saying no to it. It is really that "really understand" part which holds me back for now - mainly the observability and dealing with edge cases in high-throughput environment.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#75

Post by belmont_sup »

api wrote: post #58
qeternity wrote: post #52

These threads are always full of people who have always used an AWS/GCP/Azure service, or have never actually run the service themselves. Running HA Postgres is not easy...but at any sort of scale where this stuff matters, nothing is easy. It's not as if AWS has 100% uptime, nor is it super cheap/performant. There are tradeoffs for everyone's use-case but every thread is full of people at one end of the cloud / roll-…

I wonder how many people use things like CockroachDB, Yugabyte, or TiDB? They're at least in theory far easier to run in HA configurations at the cost of some additional overhead and in some cases more limited SQL functionality. They seem like a huge step up from the arcane "1980s Unix" nightmare of Postgres clustering but I don't hear about them that much. Are they not used much or are their users just happy and qui…

New user of cockroach. We’ll find out! If this startup ever makes it to any meaningful user sizd

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#76

Post by nunopato »

(Nhost)

Sorry for not answering everyone individually, but I see some confusion duo to the lack of context about what we do as a company.

First things first, Nhost falls into the category of backend-as-a-service. We provision and operate infrastructure at scale, and we also provide and run the necessary services for features such as user authentication and file storage, for users creating applications and businesses. A project/backend is comprised of a Postgres Database and the aforementioned services, none of it is shared. You get your own GraphQL engine, your own auth service, etc. We also provide the means to interface with the backend through our official SDKs.

Some points I see mentioned below that are worth exploring:

- One RDS instance per tenant is prohibited from a cost perspective, obviously. RDS is expensive and we have a very generous free tier.

- We run the infrastructure for thousands of projects/backends which we have absolutely no control over what they are used for. Users might be building a simple job board, or the next Facebook (please don't). This means we have no idea what the workloads and access patterns will look like.

- RDS is mature and a great product, AWS is a billion dolar company, etc - that is all true. But is it also true that we do not control if a user's project is missing an index and the fact that RDS does not provide any means to limit CPU/memory usage per database/tenant.

- We had a couple of discussions with folks at AWS and for the reasons already mentioned, there was no obvious solution to our problem. Let me reiterate this, the folks that own the service didn't have a solution to our problem given our constraints.

- Yes, this is a DIY scenario, but this is part of our core business.

I hope this clarifies some of the doubts. And I expect to have a more detailed and technical blog post about our experience soon.

By the way, we are hiring. If you think what we're doing is interesting and you have experience operating Postgres at scale, please write me an email at nuno@nhost.io. And don't forget to star us at https://github.com/nhost/nhost.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#78

Post by whoisthemachine »

kazen44 wrote: post #61
jerf wrote: post #34

Earlier quoted context omitted.

I can't blame it on "cloud", though it's not helping that there are an awful lot of cloud services that claim to be "high performance" and are often mediumish at best. But in general I see a lot of ignorance in the developer community as to how fast things should be able to run, even in terms of reading local files and doing local manipulations with no "cloud" in sight. Honestly, if I had to pin it on just one thing,…

> Few developers think about the cost of using the network. Developers do not seem to realise how slow the network is compared to everything else. Sure, 100gbit network itnerfaces do exist, but most servers are attached with 10gbit interfaces, and most of the actual implementations will not actually manage to hit something like 10gbit/s because of latency and window scaling. You cannot escape latency (without inventi…

> You cannot escape latency (without inventing another universe in which physics do not apply). And latency is detrimental to performance.

This. So few people distinguish between bandwidth and latency. One can be increased arbitrarily and fairly easily with new encoding techniques (which generally only improves edge cases), and the other has a floor that is hard-coded into our universe. I've gotten into debates with folks who think a 10GB connection from the EU to Texas should be as fast as a connection from Texas to the Midwest, or to speed up the EU-TX connection they just need to spend more on bandwidth.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#80

Post by peterangular »

sn0wf1re wrote: post #53
peterangular wrote: post #29

Earlier quoted context omitted.

I'd posit that it's not as simple. Maybe if you're just cranking out your one-off app or something of the sort... But getting a good replication setup that's HA, potentially across multiple regions/zones, all abstracted under K8s - yea. That's not trivial. And, it can go very wrong. > I bet you also hate on people making their own espresso instead of just going to starbucks This is just unnecessary.

>> I bet you also hate on people making their own espresso instead of just going to starbucks >This is just unnecessary. I agree the ad hominem is not required, although the analogy is itself decent.

I mean I can make up ad hominem analogies about this stuff too - but it practice it makes people feel attacked/defensive, and rarely ever adds nuance or context to the conversation. I feel like in this situation it could have been omitted as-per HN guidelines:

> In Comments:

> Be kind. Don't be snarky.

Post reply on HN