Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Postgres in Kubernetes [pdf]

static.sched.com

Post reply on HN
Print view View original

71–80 of 101 posts

Re: Running Postgres in Kubernetes [pdf]

#71

Post by deathanatos »

redis_mlc wrote: post #62
deathanatos wrote: post #40

Earlier quoted context omitted.

I've done MySQL RDS, and I've seen k8s database setups. (But not w/ PG.) RDS is okay, but I would not dismiss the maintenance work required; RDS puts you at the mercy of AWS when things go wrong. We had a fair bit of trouble with failovers taking 10x+ longer than they should. We also set up encryption, and that was also a PITA: we'd consistently get nodes with incorrect subjectAltNames. (Also, at the time, the certs…

I'm a DBA, so I'll do a deep dive into your comment: - RDS is awesome overall - there's no maintenance work required on your part. If you think encryption is a problem, don't use it until later. Since RDS is a managed service, I just tell compliance auditors, "It's a managed service." - Aurora had issues (um, alpha quality) for the first year or so. So don't use new databases in the first 5 years for production, as r…

> there's no maintenance work required on your part.

The post of mine you are replying to outlines maintenance work we had to do on an actual RDS instance. My point is that you shouldn't weigh managed solutions as maintenance-free: they're not (and examples of why and how they are not are in the first post). They might win out, and they do have a place, but if you're evaluating them as "hassle-free", you will be disappointed.

> If you think encryption is a problem, don't use it until later.

We had compliance requirements that required encryption, so waiting until later was not an option.

> Since RDS is a managed service, I just tell compliance auditors, "It's a managed service."

I'm not a big fan of giving compliance auditors half-truths that mislead them into thinking we're doing something we're not.

> So don't use new databases in the first 5 years for production, as recommended.

You mean we should run our own? (/s… slightly.) We were exploring Aurora as the performance of normal RDS was not sufficient. Now, there was plenty we could have done better in other area, particularly in the database schema department, but Aurora was thought to be the most pragmatic option.

Re: Running Postgres in Kubernetes [pdf]

#72

Post by chvid »

To all the commenters in this thread.

If kubernetes cannot run a database then what good is it? (And I suppose the same issues pop up for things like a persistent queue or a full text indexer.)

The end goal of Kubernetes is to able to create and recreate environments and scale them up and down at will all based a declarative configuration. But if you take databases out of it; then you are not really achieving that goal and just left with the flipside of kubernetes: a really complex setup and a piece of technology that is very hard to master.

Re: Running Postgres in Kubernetes [pdf]

#73

Post by darksaints »

nisten wrote: post #68
ianhowson wrote: post #46

Earlier quoted context omitted.

IOPS limitations and network latency are the reason I want to run my Postgres replicas in k8s. Every machine gets NVMe and a replica instance. They're still cattle, but they can serve a lot of requests. Database CPU usage is negligible. It was easier to put a replica on every host than try to rearchitect things to tolerate single-digit millisecond RTT to a cloud database.

I think you just hit upon the most overlooked factor here, storage hardware. You can get an order of magnitude of benefit from moving to NVMe. Especially when you look at nvme benchmarks using small files , they can be over 10 times faster than regular SSDs in those scenarios and have beefier firmare too. I have a strong feeling that the reason databases crap out in containers is mainly because you're likely using a…

This is the thing that keeps my databases off the cloud. No amount of managed database convenience can convince me to give up the performance of hard metal with modern NVMe SSDs. And if you really want that kind of performance in the cloud, the cloud providers are just barely starting to catch up, and it's gonna cost you dearly. I've seen managed database setups costing $3-4k per month that could be had with an $8k server.

Re: Running Postgres in Kubernetes [pdf]

#74

Post by halbritt »

sasavilic wrote: post #23

Unless you have a really good shared storage, I don't see any advantage for running Postgres in Kubernetes. Everything is more complicated without any real benefit. You can't scale it up, you can't move pod. If pg fails to start for some reason, good luck jumping into container to inspect/debug stuff. I am neither going to upgrade PG every 2 weeks nor it is my fresh new microservice that needs to be restarted when it…

I ran ~200 instances of Postgres in production for a SaaS product. This was on top of GCP persistent disk, which qualifies as quite good network storage, all of it backed up by what is now called Velero.

This particular database was not a system of record. The database stored the results of a stream processing system. In the event of a total loss of data, the database could be recovered by re-streaming the original data, making the operation of PG in a kubernetes cluster a fairly low risk endeavor. As such, HA was not implemented.

This setup has been running in production for over two years now. In spite of having no HA, each instance of this application backed by this DB had somewhere between four and five nines of availability while being continuously monitored on one minute intervals from some other spot on the Internet.

During the course of my tenure, there was only one data loss incident in which an engineer mistakenly dropped a table. Recovery was painless.

I've since moved on to another role and can't imagine having to run a database without having the benefits of Kubernetes. I'm forced to, however, as some folks aren't as progressive, and damn does it feel archaic.

Re: Running Postgres in Kubernetes [pdf]

#75

Post by sunshinekitty »

coding123 wrote: post #69
qeternity wrote: post #36

Earlier quoted context omitted.

You don't run shared/network storage. You run PVCs on local storage and you run an HA setup. You ship log files every 10s/30s/1m to object storage. You test your backups regularly which k8s is great for. All of this means that I don't worry about most things you mention. PG upgrade? Failover and upgrade the pod. Upgrade fails? wal-g clone from object storage and rejoin cluster. Scaling up? Adjust the resource claims.…

It sounds pretty simple and enticing. Any problems convincing the team of this route?

Speaking from experience..

This is fine if you have a small single node database that can have some downtime. Once you need replicas, to fit your database in ram for performance, or reliability with a hot standby for failovers it becomes a lot more complicated.

You should consider what (if anything) you miss out on by just running in a single vm that you can scale out much easier down the road should you need to. Alternatively pay extra for a hosted solution that simplifies those steps further.

Re: Running Postgres in Kubernetes [pdf]

#76

Post by kbumsik »

chvid wrote: post #72

To all the commenters in this thread. If kubernetes cannot run a database then what good is it? (And I suppose the same issues pop up for things like a persistent queue or a full text indexer.) The end goal of Kubernetes is to able to create and recreate environments and scale them up and down at will all based a declarative configuration. But if you take databases out of it; then you are not really achieving that go…

> he end goal of Kubernetes is to able to create and recreate environments and scale them up and down at will all based a declarative configuration.

PG already has its own clustering solution to scale up and down, which is orthogonal to Kubernetes. So running PG in Kubernetes does not add anything. Also, you are much more likely to mess them up when trying to mix two orthogonal technologies.

And the DB is not meant to create and recreate often unless you want to purge the data. So my take is this: Kubernetes is to manage and configure microservices and DBs are not microservices.

Re: Running Postgres in Kubernetes [pdf]

#77

Post by redis_mlc »

deathanatos wrote: post #71
redis_mlc wrote: post #62

Earlier quoted context omitted.

I'm a DBA, so I'll do a deep dive into your comment: - RDS is awesome overall - there's no maintenance work required on your part. If you think encryption is a problem, don't use it until later. Since RDS is a managed service, I just tell compliance auditors, "It's a managed service." - Aurora had issues (um, alpha quality) for the first year or so. So don't use new databases in the first 5 years for production, as r…

> there's no maintenance work required on your part. The post of mine you are replying to outlines maintenance work we had to do on an actual RDS instance. My point is that you shouldn't weigh managed solutions as maintenance-free: they're not (and examples of why and how they are not are in the first post). They might win out, and they do have a place, but if you're evaluating them as "hassle-free", you will be disa…

I bet your life is tough, being as dumb as a box of rocks.

The above answers are from somebody who doesn't know what they're talking about, either about database administration or compliance.

Re: Running Postgres in Kubernetes [pdf]

#78

Post by arronax »

caniszczyk wrote: post #10

For the MySQL folks, see Vitess as an example on how to run Kubernetes on MySQL: https://vitess.io

There are also MySQL operators from Oracle, Presslabs, and Percona. Vitess is much more than just MySQL in k8s, and not everyone will be able to switch to it easily (if at all).

Re: Running Postgres in Kubernetes [pdf]

#79

Post by jjjensen90 »

Townley wrote: post #16
mamon wrote: post #4

Earlier quoted context omitted.

It actually is not that complex. I'm using Crunchy Postgres Operator at my current employer. You get an Ansible playbook to install an Operator inside Kubernetes, and after that you get a commandline administration tool that let's you create a cluster with a simple pgo create cluster command. Most administrative tasks like creating or restoring backups (which can be automatically pushed to S3) are just one or two pgo…

I agree that the k8s ecosystem isn't quite as complex as it seems at first, but specifically running stateful apps does come pretty close to earning the bad reputation. (Disclaimer: I've tried and failed several times to get pgsql up and running in k8s with and without operators, so that either makes me unqualified to discuss this, or perfectly qualified to discuss this) If the operator were simple enough to be insta…

I run multi terabyte, billions of rows HA postgres in kubernetes using a helm chart and Patroni (baked into the chart), which uses the native k8s API for automatic failover and pgbackrest for seamlessly provisioning new replicas. It's a single helm chart and is by far the easiest DBA I've ever done in many years.

Re: Running Postgres in Kubernetes [pdf]

#80

Post by DasIch »

kgraves wrote: post #3

What's the use-case for running databases in k8s, is this a widely accepted best practice?

If you are running Kubernetes, happen to be a fairly large organization and use microservices, you probably have many databases. Hundreds of them. Most of them are going to be small, using few resources of any kind.

In that context running postgres on K8S makes a lot of sense. You already have K8S and experience running it. Running postgres there makes it possible to share resources between databases and other applications. That improves utilization which means you can reduce costs significantly.

Another advantage is that unlike managed solutions such as RDS, you can use a more recent postgres version and postgres extensions that RDS doesn't support. Extensions such as PgQ or TimescaleDB or ...

Having said all of this. In a large organization, you have the benefit of economies of scale. Large fixed costs (such as developing the expertise required to run (postgres on) K8S reliably) can be amortized. It's possible for this to be a good idea, even a best practice for large organizations while at the same time a terrible idea for smaller ones. Most of the time, using a managed service like RDS is probably a better choice. In other words: You are not Google. You are not Facebook. You are probably not even Zalando. Figure out what's right for you.

Post reply on HN