Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we moved from AWS RDS to Postgres in Kubernetes

nhost.io

Post reply on HN
Print view View original

101–110 of 153 posts

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#101

Post by Spooky23 »

CuriouslyC wrote: post #86
0xbadcafebee wrote: post #54

Earlier quoted context omitted.

What you describe is still a fallacy because it's assuming that just because you can get better performance with BareMetal, that somehow this is a cheaper or better option. In fact it will be either more error-prone, or more expensive, or both, because you are trying to reproduce from scratch what the whole RDS team has been doing for 10 years.

I don't think anyone's arguing RDS doesn't have useful features. The problem is that it's stupid expensive for the performance you get. RDS makes a lot of sense when prototyping, and if you want a failover database with checkpoint backups, but having it be your primary database of record only makes sense if you're not developing a data product, otherwise your profit margin becomes Amazon's.

RDS is to 2022 what Oracle was in 2002. It’s a safe choice.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#102

Post by jiggawatts »

kazen44 wrote: post #60
geggam wrote: post #50

Earlier quoted context omitted.

Are you talking about the cloud host to cloud host networking or the POD networking inside the single host ? The dizzying amount of NAT layers has to be killing performance. I haven't had the chance to ever sit down and unravel a system running a good load. The lack of TCP tuning combined with the required connection tracking is interesting to think about

i still dont understand why nearly all CNI's are so hell bent on implementing a dozen layers of NAT to tunnel their overlay networks, instead of implementing a proper control plane to automate it all away between routes. Calico seems to be doing it semi-okeish, and even their the control plane is kind of unfinished? The only software based solution which seem to properly have this figured out is VMware NSX-T. (i am n…

I believe Azure CNI is pretty much point-to-point.

Azure Load Balancers and their software defined network use packet header rewriting at the host level to bypass the need for the traffic to physically traverse a load balancer appliance or a NAT appliance. They're generally rewritten when they arrive to the host hypervisor. This is done in hardware via an FPGA inline with the NICs. (This requires "Accelerated Networking" to be enabled, but that's the default in v4 VMs and required for v5 VMs.)

I'm not certain, but I believe AWS does something similar for their VMs. (Their marketing material mentions that they use a custom ASIC instead of an FPGA like Azure.)

With Azure Kubernetes Service (AKS), you can use the Azure CNI, which gives each Pod a unique IP address on the Azure Virtual Network. I can't confirm, but I'm reasonably certain that this means that Pod-to-Pod traffic is direct, with no NAT appliance or software in the way. Essentially the host NICs do the address translation inline at line rate and essentially zero latency.

However, PaaS platforms like Azure App Service or Azure SQL Database are very bad in comparison. They proxy and tunnel and NAT, all in software. I've seen latencies north of 7 milliseconds within a region!

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#103

Post by [deleted] »

manfre wrote: post #95
ftufek wrote: post #62

Earlier quoted context omitted.

Honestly, that's what I initially thought trying to run ha postgres on k8s, but zalando's postgres operator made things so much easier (maybe even easier than RDS). Very easy to rollout as many postgres clusters with whatever size you want. We've been running our production db on it for the last 6 months or so, no outage yet. Though I guess if you have to have a very custom setup, it might be more difficult.

Have you tested the backup/recovery for any of the DBs yet? I'm curious to hear how that went.

[deleted]

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#104

Post by tluyben2 »

api wrote: post #58
qeternity wrote: post #52

These threads are always full of people who have always used an AWS/GCP/Azure service, or have never actually run the service themselves. Running HA Postgres is not easy...but at any sort of scale where this stuff matters, nothing is easy. It's not as if AWS has 100% uptime, nor is it super cheap/performant. There are tradeoffs for everyone's use-case but every thread is full of people at one end of the cloud / roll-…

I wonder how many people use things like CockroachDB, Yugabyte, or TiDB? They're at least in theory far easier to run in HA configurations at the cost of some additional overhead and in some cases more limited SQL functionality. They seem like a huge step up from the arcane "1980s Unix" nightmare of Postgres clustering but I don't hear about them that much. Are they not used much or are their users just happy and qui…

We are testing all our software on yugabyte now to see how well it works. The cockroach license makes it not a fit for us, so we decided to try Yuga. So far works very well for our workloads.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#105

Post by xwowsersx »

paulryanrogers wrote: post #92
xwowsersx wrote: post #2

I've recently been spending a fair amount of time trying to improve query performance on RDS. This includes reviewing and optimizing particularly nasty queries, tuning PG configuration (min_wal_size, random_page_cost, work_mem, etc). I am using a db.t3.xlarge with general purpose SSD (gp2) for a web server that sees moderate writes and a lot of reads. I know there's no real way to know other than through testing, but…

General storage IOPS scales with disk size, roughly and to a point. It's often cheaper and faster to increase the instance storage than move to EBS, prioritized or not. Of course if you need to recover quickly in a disaster you'll want a hot standby or replica. Still may be cheaper than PIOPs. (Especially if you need HA anyway.)

Thanks for the reply.

> It's often cheaper and faster to increase the instance storage than move to EBS, prioritized or not.

You're saying it may well be cheaper to increase storage in order to get more IOPS than moving to an EBS-optimized instance type?

Regarding HA, not relevant for at this point (assuming I understood you correctly). We've only got a single primary and one replica, the latter being used primarily for analytics.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#106

Post by xwowsersx »

brazzledazzle wrote: post #87
notac wrote: post #3

Earlier quoted context omitted.

I'd recommend hopping off of t3 asap if you're searching for performance gains - performance can be extremely variable (by design). M class will even you out. General storage IOPS is governed by your provisioned storage size. You can again get much more consistent performance by using provisioned IOPS. Feel free to email me if you want to chat through things specific to your env - email is in my about:

Provisioned IOPS is much more expensive though, so make sure you really need it. If you use general IOPS you can monitor your burst balance. You can always start with general and then move to provisioned when you need it too.

Thanks. If I'm reading this https://ibb.co/bNGmrCB correctly, it seems like we have plenty burst balance. Does this seem to indicate that provisioned IOPS is unlikely to help us here?

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#107

Post by osigurdson »

kazen44 wrote: post #61
jerf wrote: post #34

Earlier quoted context omitted.

I can't blame it on "cloud", though it's not helping that there are an awful lot of cloud services that claim to be "high performance" and are often mediumish at best. But in general I see a lot of ignorance in the developer community as to how fast things should be able to run, even in terms of reading local files and doing local manipulations with no "cloud" in sight. Honestly, if I had to pin it on just one thing,…

> Few developers think about the cost of using the network. Developers do not seem to realise how slow the network is compared to everything else. Sure, 100gbit network itnerfaces do exist, but most servers are attached with 10gbit interfaces, and most of the actual implementations will not actually manage to hit something like 10gbit/s because of latency and window scaling. You cannot escape latency (without inventi…

Light travels 300km in 1 millisecond. Intra datacenter latency is not bounded by physics. It is bounded by current technology.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#108

Post by remram »

brand wrote: post #30
radimm wrote: post #4

Having recently heard a lot of about PostgreSQL in Kubernetes (cloudNativePG for example) it always makes me wonder about the actual load and the complexity of the cluster in the question. > This is the reason why we were able to easily cope with 2M+ requests in less than 24h when Midnight Society launched This gives the answer, while it's probably not evenly distributed gives 23 req/sec (guess peak 60 - 100 might be…

I’ve personally deployed O(TBs) and O(10^4 TPS) Postgres clusters on Kubernetes with a CNPG-style operator based deployment. There are some subtleties to it but it’s not exceeding complicated, and a good project like CNPG goes a long way to shaving off those sharp edges. As other commenters have suggested it’s good to really understand Kubernetes if you want to do it, though.

> O(TBs) and O(10^4 TPS)

What does this syntax mean? Surely you wouldn't use big-o notation with a constant in it, especially to convey the same meaning as the thing without the O?

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#109

Post by dilyevsky »

kazen44 wrote: post #61
jerf wrote: post #34

Earlier quoted context omitted.

I can't blame it on "cloud", though it's not helping that there are an awful lot of cloud services that claim to be "high performance" and are often mediumish at best. But in general I see a lot of ignorance in the developer community as to how fast things should be able to run, even in terms of reading local files and doing local manipulations with no "cloud" in sight. Honestly, if I had to pin it on just one thing,…

> Few developers think about the cost of using the network. Developers do not seem to realise how slow the network is compared to everything else. Sure, 100gbit network itnerfaces do exist, but most servers are attached with 10gbit interfaces, and most of the actual implementations will not actually manage to hit something like 10gbit/s because of latency and window scaling. You cannot escape latency (without inventi…

Directly attached NVME drives will have throughput of up to 30-50Gbps (with something like m.2) which should be attainable with NVMe-oF over QSFP28 and it's not that rare or expensive anymore. Others have commented on the latency. Fibre Channel can be considered network too (and it is) and it's quite fast.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#110

Post by hotpotamus »

osigurdson wrote: post #107
kazen44 wrote: post #61

Earlier quoted context omitted.

> Few developers think about the cost of using the network. Developers do not seem to realise how slow the network is compared to everything else. Sure, 100gbit network itnerfaces do exist, but most servers are attached with 10gbit interfaces, and most of the actual implementations will not actually manage to hit something like 10gbit/s because of latency and window scaling. You cannot escape latency (without inventi…

Light travels 300km in 1 millisecond. Intra datacenter latency is not bounded by physics. It is bounded by current technology.

And in 1 millisecond, a 1Ghz CPU will have 1 million cycles. It's a bit like sending a letter and then waiting a month or two for a response.
Post reply on HN