Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we moved from AWS RDS to Postgres in Kubernetes

nhost.io

Post reply on HN
Print view View original

131–140 of 153 posts

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#131

Post by sgtfrankieboy »

api wrote: post #58
qeternity wrote: post #52

These threads are always full of people who have always used an AWS/GCP/Azure service, or have never actually run the service themselves. Running HA Postgres is not easy...but at any sort of scale where this stuff matters, nothing is easy. It's not as if AWS has 100% uptime, nor is it super cheap/performant. There are tradeoffs for everyone's use-case but every thread is full of people at one end of the cloud / roll-…

I wonder how many people use things like CockroachDB, Yugabyte, or TiDB? They're at least in theory far easier to run in HA configurations at the cost of some additional overhead and in some cases more limited SQL functionality. They seem like a huge step up from the arcane "1980s Unix" nightmare of Postgres clustering but I don't hear about them that much. Are they not used much or are their users just happy and qui…

We have multiple CockroachDB clusters, have been for 4+ years now. From 2TB to 14TB in used size, the largest does about 3k/req sec.

We run them on dedicated hosts or on Hetzner cloud instances. We tested out RDS Postgres, but that would've literally tripled our cost for worse performance.

Only had a few hiccups with the big cluster but they were resolved quickly with their support.

We're very happy with the product, and have leaned quite a few optimization tricks to get the best out of it. Easy to use as well, join the nodes and it just works.

It's not perfect though, we've had quite a few issues with deleting lots of data at once, it doesn't like that. So we have to do deleted in smaller chunks.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#133

Post by ants_a »

manfre wrote: post #95
ftufek wrote: post #62

Earlier quoted context omitted.

Honestly, that's what I initially thought trying to run ha postgres on k8s, but zalando's postgres operator made things so much easier (maybe even easier than RDS). Very easy to rollout as many postgres clusters with whatever size you want. We've been running our production db on it for the last 6 months or so, no outage yet. Though I guess if you have to have a very custom setup, it might be more difficult.

Have you tested the backup/recovery for any of the DBs yet? I'm curious to hear how that went.

Setup credentials for an S3-compatible bucket and you will have backups + point-in-time-recovery capability. For restores there are a few different options. If you just flat out delete the persistent volumes you will get an automatic restore to latest state in the archive (RPO of ~16MB of transaction log in case of unclean shutdown). Or if you just need to recover some data from a previous state, you can clone a new cluster from backups specifying a timestamp to recover to. Or in case you strictly need to get the same cluster up to a previous state, you can manually exec into the container stick it into maintenance mode and replace the contents with a backup.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#134

Post by xyzzy_plugh »

movedx wrote: post #123
xyzzy_plugh wrote: post #118

If the cost of operating a postgres database is eating into your margins so much (and you can't simply adjust your prices to eat the difference) then I would suspect the wrong technology is in place. Sure, RDS is expensive, but it's also quite well done. Almost every cloud platform service is more expensive than doing it yourself. No surprise here. In the past I've deployed SQLite over Postgres for cost cutting reaso…

> Almost every cloud platform service is more expensive than doing it yourself. No surprise here. In a business environment, this is actually not true unless you consider the extreme long term. A Multi-AZ MySQL RDS instance of size db.m1.large (2x vCPU, 7.5GB of RAM), a 500GB standard disk, and an on-demand pricing model with 100% monthly utilization, will cost you approx. US$7,000 per year (rounding up.) That price…

Agreed. RDS is a no-brainer, generally. The issue here appears to be with the unit economics per tenant. My argument is if the unit economics matter so much, the technology choice is likely a poor fit.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#135

Post by osigurdson »

ahachete wrote: post #124
osigurdson wrote: post #107

Earlier quoted context omitted.

Light travels 300km in 1 millisecond. Intra datacenter latency is not bounded by physics. It is bounded by current technology.

Light travels much slower (~1.5x slower) on a fiber optic, due to the refractive index (~ 1.5) of the fiber.

Yes, but that is not even the lowest latency technology available today.

https://en.wikipedia.org/wiki/Velocity_factor

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#136

Post by tbran »

mhuffman wrote: post #28
Nextgrid wrote: post #7

Earlier quoted context omitted.

> 23 req/sec (guess peak 60 - 100 might be already stretching it) That kind of load is something a decent developer laptop with an NVME drive can serve, nothing to write home about. It is sad that the "cloud" and all these supposedly "modern" DevOps systems managed to redefine the concept of "performance" for a large chunk of the industry.

It does depend on the architecture and framework they are using imo. I have a single Hetzer machine with spinning plate HDs that serves between 1-2 million requests per day hitting DB and ML models and rarely every gets over 1% CPU usage. I have pressure-tested it to around 3k reqs/sec. On the other hand I have seen WP and CodeIgniter setups that even with 5 copies running on the largest AWS instances available, "opt…

Can you clarify what you mean by "tons of unnecessary text in the DB"?

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#137

Post by dns_snek »

hotpotamus wrote: post #110
osigurdson wrote: post #107

Earlier quoted context omitted.

Light travels 300km in 1 millisecond. Intra datacenter latency is not bounded by physics. It is bounded by current technology.

And in 1 millisecond, a 1Ghz CPU will have 1 million cycles. It's a bit like sending a letter and then waiting a month or two for a response.

Nitpick: 1 billion, not 1 million ;)

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#138

Post by remram »

MikePlacid wrote: post #119
remram wrote: post #108

Earlier quoted context omitted.

> O(TBs) and O(10^4 TPS) What does this syntax mean? Surely you wouldn't use big-o notation with a constant in it, especially to convey the same meaning as the thing without the O?

Mathematically speaking the statement you are objecting to is correct: c1 is O(c2) for any constants c1, c2. English-language-ly speaking the statement you are objecting to is also correct: both you and I managed to get it’s correct meaning. No?

No, I am not sure about the meaning at all. Did they deploy databases that will tend to be a TB in size as something tends to infinity? Or multiple TBs? I don't know if they know about the constant factor since they don't know what the notation mean. Maybe they know what it means and are using it for a clever lie, a 1kB database is O(1TB). So is an empty database.

GGP is trying to be cool, and doing so stripped all meaning from their statement.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#139

Post by osigurdson »

dns_snek wrote: post #137
hotpotamus wrote: post #110

Earlier quoted context omitted.

And in 1 millisecond, a 1Ghz CPU will have 1 million cycles. It's a bit like sending a letter and then waiting a month or two for a response.

Nitpick: 1 billion, not 1 million ;)

THz?

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#140

Post by CodesInChaos »

xwowsersx wrote: post #21
Nextgrid wrote: post #17

Earlier quoted context omitted.

You've got some spikes that could signify some large or unoptimized queries, but otherwise yes, the CPU doesn't look that hot. I suggest upgrading to an instance type which gives you 32GB or more of memory. You'll get a bigger CPU along with it as well, but don't make the CPU your priority, it's not your main bottleneck at the moment.

Makes sense, thank you. Sounds like M class is the way to go as other commenter suggested. Also, yes. There are many awful queries that I'm aware of and working to correct.

I'd look into Memory Optimized (R*) instances.
Post reply on HN