Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we moved from AWS RDS to Postgres in Kubernetes

nhost.io

Post reply on HN
Print view View original

51–60 of 153 posts

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#51

Post by deathanatos »

wbl wrote: post #27
xani_ wrote: post #23

Earlier quoted context omitted.

Quoted post unavailable.

Running a statefull service in K8S is its own ball of wax

… I do it, in my day job. It's really not. StatefulSets are explicitly for this.

We also have managed databases, too.

Self-managed stuff means I can, generally, get shit done with it, when oddball things need doing. Managed stuff is fine right up until it isn't (i.e., yet another outage with the status page being green), or until there's a requirement that the managed system inexplicably can't handle (despite the requirement being the sort of obvious thing you would expect of $SYSTEM, but which no PM thought to ask before purchasing the deal…), and then you're in support ticket hell.

(E.g., we found out the hard way that there is not way to move a managed PG database from one subnet in a network to another, in Azure! Even if you're willing to restore from a backup. We had to deal with that ourselves, by taking a pgdump — essentially, un-managed-solution the backup.

… the whole reason we needed to move the DB to a different subnet was because of a different flaw, in a different managed service, and Azure's answer on that ticket was "tough luck DB needs to move". Tickets, spawning tickets. Support tickets for managed services take up an unholy portion of my time.)

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#52

Post by qeternity »

These threads are always full of people who have always used an AWS/GCP/Azure service, or have never actually run the service themselves.

Running HA Postgres is not easy...but at any sort of scale where this stuff matters, nothing is easy. It's not as if AWS has 100% uptime, nor is it super cheap/performant. There are tradeoffs for everyone's use-case but every thread is full of people at one end of the cloud / roll-your-own spectrum.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#53

Post by sn0wf1re »

peterangular wrote: post #29
xani_ wrote: post #23

Earlier quoted context omitted.

Quoted post unavailable.

I'd posit that it's not as simple. Maybe if you're just cranking out your one-off app or something of the sort... But getting a good replication setup that's HA, potentially across multiple regions/zones, all abstracted under K8s - yea. That's not trivial. And, it can go very wrong. > I bet you also hate on people making their own espresso instead of just going to starbucks This is just unnecessary.

>> I bet you also hate on people making their own espresso instead of just going to starbucks

>This is just unnecessary.

I agree the ad hominem is not required, although the analogy is itself decent.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#54

Post by 0xbadcafebee »

suggala wrote: post #26
0xbadcafebee wrote: post #6

Ah, the 'ol sunk cost fallacy of infrastructure. We are already investing in supporting K8s, so let's throw the databases in there too. Couldn't possibly be that much work. Sure, a decade-old dedicated team at a billion-dollar multinational corporation has honed a solution designed to support hundreds of thousands of customers with high availability, and we could pay a little bit extra money to spin up a new database…

AWS RDS is 10x slower than BareMetal MySQL (both reads and writes). Slowness is mainly due to the reason that Storage is over network for RDS. Not bad to invest some extra time to get better performance. You are falling to “Appeal to antiquity” fallacy if you think something old is better.

What you describe is still a fallacy because it's assuming that just because you can get better performance with BareMetal, that somehow this is a cheaper or better option. In fact it will be either more error-prone, or more expensive, or both, because you are trying to reproduce from scratch what the whole RDS team has been doing for 10 years.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#55

Post by marcosdumay »

Nextgrid wrote: post #33
mhuffman wrote: post #28

Earlier quoted context omitted.

It does depend on the architecture and framework they are using imo. I have a single Hetzer machine with spinning plate HDs that serves between 1-2 million requests per day hitting DB and ML models and rarely every gets over 1% CPU usage. I have pressure-tested it to around 3k reqs/sec. On the other hand I have seen WP and CodeIgniter setups that even with 5 copies running on the largest AWS instances available, "opt…

Keep in mind that your Hetzner instance has locally-attached storage and a real CPU as opposed to networked storage and a slice of a CPU, so I'm not surprised at all that this beats an AWS setup even on the more expensive instances. Yes, frameworks can be a problem (although including WP in the list is an insult to other, actually decent frameworks), but I would bet good money if they moved their setup to a Hetzner s…

Hum... The Hetzner instance is very likely cheaper than any AWS setup, so while there is a point in that part, it's not a very relevant one. (And that's exactly the issue with the "modern DevOps" tooling.)

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#56

Post by neilv »

I didn't see "backups" mentioned in that, though I'm sure they have them. Depending on your needs, it's a big thing to keep in mind while weighing options.

For a small startup or operation, a managed service having credible snapshots, PITR backups, failover, etc. is going to save a business a lot of ops cost, compared to DIY designing, implementing, testing, and drilling, to the same level of credibility.

One recent early startup, I looked at the amount of work for me or a contractor/consultant/hire to upgrade our Postgres recovery capability (including testing and drills) with confidence. I soon decided to move from self-hosted Postgres to RDS Postgres.

RDS was a significant chunk of our modest AWS bill (otherwise, almost entirely plain EC2, S3, and traffic), but easy to justify to the founders, just by mentioning the costs it saved us for business existential protection we needed.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#57

Post by janee »

mijamo wrote: post #48
patrec wrote: post #38

Earlier quoted context omitted.

It is, but then I never understood why on earth you'd use k8s if you don't have stateful services. I mean really, what's the point?

Because it's easy? What alternative would you suggest?

Bare metal I'd think is the first choice for a large rdbms where you have skilled dedicated personnel that can manage it.

If not rather use a specialist service like RDS for anything with serious uptime/throughput requirements.

k8s doesn't really make sense to me unless it's for spinning up lots of instances, like for test or dev envs or like in the article where they host DBs for people.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#58

Post by api »

qeternity wrote: post #52

These threads are always full of people who have always used an AWS/GCP/Azure service, or have never actually run the service themselves. Running HA Postgres is not easy...but at any sort of scale where this stuff matters, nothing is easy. It's not as if AWS has 100% uptime, nor is it super cheap/performant. There are tradeoffs for everyone's use-case but every thread is full of people at one end of the cloud / roll-…

I wonder how many people use things like CockroachDB, Yugabyte, or TiDB? They're at least in theory far easier to run in HA configurations at the cost of some additional overhead and in some cases more limited SQL functionality.

They seem like a huge step up from the arcane "1980s Unix" nightmare of Postgres clustering but I don't hear about them that much. Are they not used much or are their users just happy and quiet?

(These are all "NewSQL" databases.)

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#59

Post by c2h5oh »

Nextgrid wrote: post #7
radimm wrote: post #4

Having recently heard a lot of about PostgreSQL in Kubernetes (cloudNativePG for example) it always makes me wonder about the actual load and the complexity of the cluster in the question. > This is the reason why we were able to easily cope with 2M+ requests in less than 24h when Midnight Society launched This gives the answer, while it's probably not evenly distributed gives 23 req/sec (guess peak 60 - 100 might be…

> 23 req/sec (guess peak 60 - 100 might be already stretching it) That kind of load is something a decent developer laptop with an NVME drive can serve, nothing to write home about. It is sad that the "cloud" and all these supposedly "modern" DevOps systems managed to redefine the concept of "performance" for a large chunk of the industry.

That kind of a load you can handle on spinning rust without breaking a sweat.

Re: Why we moved from AWS RDS to Postgres in Kubernetes

#60

Post by kazen44 »

geggam wrote: post #50
jerf wrote: post #34

Earlier quoted context omitted.

I can't blame it on "cloud", though it's not helping that there are an awful lot of cloud services that claim to be "high performance" and are often mediumish at best. But in general I see a lot of ignorance in the developer community as to how fast things should be able to run, even in terms of reading local files and doing local manipulations with no "cloud" in sight. Honestly, if I had to pin it on just one thing,…

Are you talking about the cloud host to cloud host networking or the POD networking inside the single host ? The dizzying amount of NAT layers has to be killing performance. I haven't had the chance to ever sit down and unravel a system running a good load. The lack of TCP tuning combined with the required connection tracking is interesting to think about

i still dont understand why nearly all CNI's are so hell bent on implementing a dozen layers of NAT to tunnel their overlay networks, instead of implementing a proper control plane to automate it all away between routes.

Calico seems to be doing it semi-okeish, and even their the control plane is kind of unfinished?

The only software based solution which seem to properly have this figured out is VMware NSX-T. (i am not counting all the traditional overlay networks in use by ISP's based on MPLS/BGP).

Post reply on HN