Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go ahead, self-host Postgres

pierce.dev

Post reply on HN
Print view View original

201–210 of 407 posts

Re: Go ahead, self-host Postgres

#201

Post by tempest_ »

j45 wrote: post #198
paulryanrogers wrote: post #160

Earlier quoted context omitted.

Disks go bad. RAID is nontrivial to set up. Hetzner had a big DC outage that lead to data loss. Off site backups or replication would help, though not always trivial to fail over.

Not as often as you might think. Hardware doesn’t fail like it used to. Hardware also monitors itself reasonably well because the hosting providers use it. It’s trivial to run a mirrored containers on two separate proxmox nodes because hosting providers use the same kind of stuff. Offsite backups and replication? Also point and click and trivial with tools like Proxmox. RAID is actually trivial to setup.l if you don’…

> if you don’t compare it to doing it manually yourself

Even if you do ZFS makes this pretty trivial as well.

Re: Go ahead, self-host Postgres

#202

Post by ninkendo »

Nextgrid wrote: post #185
mystifyingpoi wrote: post #168

Earlier quoted context omitted.

Docker Compose (ignoring Swarm which seems to be obsolete) manages containers on a single machine. With Kubernetes, the pod that hosts the database is a pod like any other (I assume). It gets moved to a healthy machine when node goes bad, respects CPU/mem limits, works with generic monitoring tools, can be deployed from GitOps tools etc. All the k8s goodies apply.

When it comes to a DB moving the process around is easy, it's the data that matters. The reason bare-metal-hosted DBs are so fast is that they use direct-attach storage instead of networked storage. You lose those speed advantages if you move to distributed storage (Ceph/etc).

You don’t need to use networked storage, the zalando postgres operator just uses local storage on the host. It uses a StatefulSet underneath so that pods will stay on the same node until you migrate them.

Re: Go ahead, self-host Postgres

#203

Post by starttoaster »

Thaxll wrote: post #197
mittermayr wrote: post #116

Self-hosting is more a question of responsibility I'd say. I am running a couple of SaaS products and self-host at much better performance at a fraction of the cost of running this on AWS. It's amazing and it works perfectly fine. For client projects, however, I always try and sell them on paying the AWS fees, simply because it shifts the responsibility of the hardware being "up" to someone else. It does not inherent…

That argument does not hold when there is aws serverless pg available, which cost almost nothing for low traffic and is vastly superior to self hosting regarding observability, security, integration, backup ect... There is no reason to self manage pg for dev / environnement. https://aws.amazon.com/rds/aurora/serverless/

"which cost almost nothing for low traffic" you invented the retort "what about high traffic" within your own message. I don't even necessarily mean user traffic either. But if you constantly have to sync new records over (as could be the case in any kind of timeseries use-case) the internal traffic could rack up costs quickly.

"vastly superior to self hosting regarding observability" I'd suggest looking into the cnpg operator for Postgres on Kubernetes. The builtin metrics and official dashboard is vastly superior to what I get from Cloudwatch for my RDS clusters. And the backup mechanism using Barman for database snapshots and WAL backups is vastly superior to AWS DMS or AWS's disk snapshots which aren't portable to a system outside of AWS if you care about avoiding vendor lock-in.

Re: Go ahead, self-host Postgres

#204

Post by mattmanser »

paulryanrogers wrote: post #160
mattmanser wrote: post #136

Earlier quoted context omitted.

Over 20 year I've had lots of clients on self-hosted, even self-hosting SQL on the same VM as the webserver as you used to in the long distant past for low-usage web apps. I have never, ever, ever had a SQL box go down. I've had a web server go down once. I had someone who probably shouldn't have had access to a server accidentally turn one off once. The only major outage I've had (2/3 hours) was when the box was als…

Disks go bad. RAID is nontrivial to set up. Hetzner had a big DC outage that lead to data loss. Off site backups or replication would help, though not always trivial to fail over.

So can the cloud, and cloud has had more major outages in the last 3 months than I've seen on self-hosted in 20 years.

Deploys these days take minutes so what's the problem if a disk does go bad? You lose at most a day of data if you go with the 'standard' overnight backups, and if it's mission critical, you will have already set up replicas, which again is pretty trivial and only slightly more complicated than doing it on cloud hosts.

Re: Go ahead, self-host Postgres

#205

Post by lofaszvanitt »

I was on a severely restricted budget and self hosted everything for 15+ years, while the heavily used part of the database was on a RAM card. The RAM drive was soft raided to a hard drive pair which were 3Ware raid1 hdds, just in case, and also did a daily backup on the database and during that time never had any data loss and never had to restore anything from backup. And my options were severely restricted due to a capped income.

The real downside wasn't technical. The constant background anxiety you had to learn to live with, since the hosted news sites were hammered by the users. The dreaded SMS alerts saying the server was inaccessible (often due to ISP issues) or going abroad meant persuading one of your mates to keep an eye on things just in case, created a lot of unnecessary stress.

AWS is quite good. It has everything you need and removes most of that operational burden, so the angst is much lower, but the pricing is problematic.

Re: Go ahead, self-host Postgres

#206

Post by reilly3000 »

I think we can get to the point where we have self-hosted agents that can manage db maintenance and recovery. There could be regular otel -> * -> Grafana -> ~PagerDuty -> you and TriageBot which would call specialists to gather state and orchestrate a response.

Scripts could kick off health reports and trigger operations. Upgrades and recovery runbooks would be clearly defined and integration tested.

It would empower personal sovereignty.

Someone should make this in the open. Maybe it already exists, there are a lot of interesting agentops projects.

If that worked 60% of the time and I had to figure out the rest, I’d self host that. I’d pay for 80%+.

Re: Go ahead, self-host Postgres

#207

Post by SatvikBeri »

emseetech wrote: post #103
molf wrote: post #36

> I'd argue self-hosting is the right choice for basically everyone, with the few exceptions at both ends of the extreme: > If you're just starting out in software & want to get something working quickly with vibe coding, it's easier to treat Postgres as just another remote API that you can call from your single deployed app > If you're a really big company and are reaching the scale where you need trained database e…

| self hosting costs you between 30 and 120 minutes per month Can we honestly say that cloud services taking a half hour to two hours a month of someone's time on average is completely unheard of?

I handle our company's RDS instances, and probably spend closer to 2 hours a year than 2 hours a month over the last 8 years.

It's definitely expensive, but it's not time-consuming.

Re: Go ahead, self-host Postgres

#208

Post by gopher_space »

zbentley wrote: post #135
jbverschoor wrote: post #129

Earlier quoted context omitted.

Yea I agree.. better outsource product development, management, and everything else too by that narrative

That’s pretty reductive. By that logic the opposite extreme is just as true: if using managed services is just as bad as outsourcing everything else, then a business shouldn’t rent real estate either—every business should build and own their own facility. They should also never contract out janitorial work, nor should they retain outside law firms—they should hire and staff those departments internally, every time, n…

I'll be reductive in conversations like this just to help push the pendulum back a little. The prevailing attitude seems (to me) like people find self-hosting mystical and occult, yet there's never been a better time to do it.

> But it’s very case-by-case. There’s no general rule like “always prefer self hosting” or “always rent real estate, never buy” that applies broadly enough to be useful.

I don't know if anyone remembers that irritating "geek code" thing we were doing a while back, but coming up with some kind of shorthand for whatever context we're talking about would be useful.

Re: Go ahead, self-host Postgres

#209

Post by Nextgrid »

ninkendo wrote: post #202
Nextgrid wrote: post #185

Earlier quoted context omitted.

When it comes to a DB moving the process around is easy, it's the data that matters. The reason bare-metal-hosted DBs are so fast is that they use direct-attach storage instead of networked storage. You lose those speed advantages if you move to distributed storage (Ceph/etc).

You don’t need to use networked storage, the zalando postgres operator just uses local storage on the host. It uses a StatefulSet underneath so that pods will stay on the same node until you migrate them.

But if I'm pinning it to dedicated machines then Kubernetes does not give me anything, but I still have to deal with its tradeoffs and moving parts - which from experience are more likely to bring me down than actual hardware failure.

Re: Go ahead, self-host Postgres

#210

Post by briHass »

Nextgrid wrote: post #83
convolvatron wrote: post #40

Earlier quoted context omitted.

its not. I've been in a few shops that use RDS because they think their time is better spend doing other things. except now they are stuck trying to maintain and debug Postgres without having the same visibility and agency that they would if they hosted it themselves. situation isn't at all clear.

One thing unaccounted for if you've only ever used cloud-hosted DBs is just how slow they are compared to a modern server with NVME storage. This leads the developers to do all kinds of workarounds and reach for more cloud services (and then integrating them and - often poorly - ensuring consistency across them) because the cloud hosted DB is not able to handle the load. On bare-metal, you can go a very long way with…

This is the reason I manage SQL Server on a VM in Azure instead of their PaaS offering. The fully managed SQL has terrible performance unless you drop many thousands a month. The VM I built is closer to 700 a month.

Running on IaaS also gives you more scalability knobs to tweak: SSD Iops and b/w, multiple drives for logs/partitions, memory optimized VMs, and there's a lot of low level settings that aren't accessible in managed SQL. Licensing costs are also horrible with managed SQL Server, where it seems like you pay the Enterprise level, but running it yourself offers lower cost editions like Standard or Web.

Post reply on HN