Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we moved from AWS RDS to Postgres in Kubernetes

nhost.io

Post reply on HN
Print view View original

61–70 of 153 posts

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#61

Post by kazen44 »

jerf wrote: post #34
Nextgrid wrote: post #7

Earlier quoted context omitted.

> 23 req/sec (guess peak 60 - 100 might be already stretching it) That kind of load is something a decent developer laptop with an NVME drive can serve, nothing to write home about. It is sad that the "cloud" and all these supposedly "modern" DevOps systems managed to redefine the concept of "performance" for a large chunk of the industry.

I can't blame it on "cloud", though it's not helping that there are an awful lot of cloud services that claim to be "high performance" and are often mediumish at best. But in general I see a lot of ignorance in the developer community as to how fast things should be able to run, even in terms of reading local files and doing local manipulations with no "cloud" in sight. Honestly, if I had to pin it on just one thing,…

> Few developers think about the cost of using the network.

Developers do not seem to realise how slow the network is compared to everything else.

Sure, 100gbit network itnerfaces do exist, but most servers are attached with 10gbit interfaces, and most of the actual implementations will not actually manage to hit something like 10gbit/s because of latency and window scaling.

You cannot escape latency (without inventing another universe in which physics do not apply). And latency is detrimental to performance.

Getting anything across a large enough network under 1millisecond is hard, and compared to a IOP on a local NVME disk, it is painfully slow.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#62

Post by ftufek »

qeternity wrote: post #52

These threads are always full of people who have always used an AWS/GCP/Azure service, or have never actually run the service themselves. Running HA Postgres is not easy...but at any sort of scale where this stuff matters, nothing is easy. It's not as if AWS has 100% uptime, nor is it super cheap/performant. There are tradeoffs for everyone's use-case but every thread is full of people at one end of the cloud / roll-…

Honestly, that's what I initially thought trying to run ha postgres on k8s, but zalando's postgres operator made things so much easier (maybe even easier than RDS). Very easy to rollout as many postgres clusters with whatever size you want. We've been running our production db on it for the last 6 months or so, no outage yet. Though I guess if you have to have a very custom setup, it might be more difficult.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#63

Post by baq »

dijit wrote: post #11
0xbadcafebee wrote: post #6

Ah, the 'ol sunk cost fallacy of infrastructure. We are already investing in supporting K8s, so let's throw the databases in there too. Couldn't possibly be that much work. Sure, a decade-old dedicated team at a billion-dollar multinational corporation has honed a solution designed to support hundreds of thousands of customers with high availability, and we could pay a little bit extra money to spin up a new database…

And when it all goes bottoms up it will be much more difficult to resolve.

Fortunately Postgres doesn’t do that often by itself. It usually needs some creative developer’s assistance.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#64

Post by dijit »

baq wrote: post #63
dijit wrote: post #11

Earlier quoted context omitted.

And when it all goes bottoms up it will be much more difficult to resolve.

Fortunately Postgres doesn’t do that often by itself. It usually needs some creative developer’s assistance.

I think you’re triggering the worst case a lot more often when it comes to running Postgres on k8s: the storage can be removed independently from the workload and the pod can be evicted much easier than it would be in traditional database hosting methods.

No need for developers to do anything strange at all.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#66

Post by patrec »

mijamo wrote: post #48
patrec wrote: post #38

Earlier quoted context omitted.

It is, but then I never understood why on earth you'd use k8s if you don't have stateful services. I mean really, what's the point?

Because it's easy? What alternative would you suggest?

The idea that something of the monstrous complexity of k8s is easy is pretty funny to me. I think if you have less than than 2 full time experts on k8s at hand, you're basically nuts if you use it for some non-toy project. In my experience, you can and will experience interesting failure scenarios.

If you don't have state, why not just either use something serverless/fully-managed (beanstalk, lambda, cloudflare workers whatever) if you really need to scale up and down (or have very limited devops/sysadmin capacity) or deploy like 2 or 3 bare metal machines or VMs?

Either sounds like a lot less work to manage and troubleshoot than some freaking k8s cluster.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#67

Post by AccountAccount1 »

rrampage wrote: post #9
Nextgrid wrote: post #7

Earlier quoted context omitted.

> 23 req/sec (guess peak 60 - 100 might be already stretching it) That kind of load is something a decent developer laptop with an NVME drive can serve, nothing to write home about. It is sad that the "cloud" and all these supposedly "modern" DevOps systems managed to redefine the concept of "performance" for a large chunk of the industry.

It depends a lot on the backend architecture. Number of DB requests per web request can also be high due to the pathological cases in some ORMs which can result in N+1 query problems or eagerly fetching entire object hierarchies. Such problems in application code can get brushed under the carpet due to "magical" autoscaling (be it RDS or K8s). There can also be fanout to async services/job queues which will in turn r…

Hey, this is not a problem for us at Nhost since most of the interfacing with Postgres is through Hasura (a GraphQL SQL-to-GraphQL) it solves the n+1 issue by compiling a performant sql statement from the gql query (it's also written in haskell, you can read more here https://hasura.io/blog/architecture-of-a-high-performance-gr...)

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#68

Post by geggam »

kazen44 wrote: post #60
geggam wrote: post #50

Earlier quoted context omitted.

Are you talking about the cloud host to cloud host networking or the POD networking inside the single host ? The dizzying amount of NAT layers has to be killing performance. I haven't had the chance to ever sit down and unravel a system running a good load. The lack of TCP tuning combined with the required connection tracking is interesting to think about

i still dont understand why nearly all CNI's are so hell bent on implementing a dozen layers of NAT to tunnel their overlay networks, instead of implementing a proper control plane to automate it all away between routes. Calico seems to be doing it semi-okeish, and even their the control plane is kind of unfinished? The only software based solution which seem to properly have this figured out is VMware NSX-T. (i am n…

Before you even get to the CNI, I think AWS VM to internet is at least 3 NAT layers.

So we have 3 layers from container to pod. The virtual host kernel is tracking those layers. Once connection to one container is 3 tracked connections. Then you have whatever else you put on top to go in and out of the internet.

The funny think to me is HaProxy recommended getting rid of connection tracking for performance while everyone is doubling down on that alone and calling it performant.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#69

Post by acdha »

mhuffman wrote: post #28
Nextgrid wrote: post #7

Earlier quoted context omitted.

> 23 req/sec (guess peak 60 - 100 might be already stretching it) That kind of load is something a decent developer laptop with an NVME drive can serve, nothing to write home about. It is sad that the "cloud" and all these supposedly "modern" DevOps systems managed to redefine the concept of "performance" for a large chunk of the industry.

It does depend on the architecture and framework they are using imo. I have a single Hetzer machine with spinning plate HDs that serves between 1-2 million requests per day hitting DB and ML models and rarely every gets over 1% CPU usage. I have pressure-tested it to around 3k reqs/sec. On the other hand I have seen WP and CodeIgniter setups that even with 5 copies running on the largest AWS instances available, "opt…

> On the other hand I have seen WP and CodeIgniter setups that even with 5 copies running on the largest AWS instances available, "optimized" to the hilt, caching everywhere possible, etc. absolute crumble under the load of 3k req per min. (not sec ... min).

This sounds like some other architectural problems - running nowhere near the largest instances available that was single node performance on EC2 in the 2000s.

There are concerns switching from local to SAN storage, of course, but that’s also shifting the problem if you care about durability.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#70

Post by ayende »

Nextgrid wrote: post #7
radimm wrote: post #4

Having recently heard a lot of about PostgreSQL in Kubernetes (cloudNativePG for example) it always makes me wonder about the actual load and the complexity of the cluster in the question. > This is the reason why we were able to easily cope with 2M+ requests in less than 24h when Midnight Society launched This gives the answer, while it's probably not evenly distributed gives 23 req/sec (guess peak 60 - 100 might be…

> 23 req/sec (guess peak 60 - 100 might be already stretching it) That kind of load is something a decent developer laptop with an NVME drive can serve, nothing to write home about. It is sad that the "cloud" and all these supposedly "modern" DevOps systems managed to redefine the concept of "performance" for a large chunk of the industry.

You are off by a couple of orders of magnitude

I have run 500+ req/sec on a raspberry pi using 4 TB dataset with 2 GB of RAM, with under 100ms for the 99.99 percentile

A few hundreds req a second is basically nothing.

Post reply on HN