Ask HN: Do you self-host your database? • Hacker News Board

Re: Ask HN: Do you self-host your database?

#81

Post by tester34 » Tue, Jun 29, 2021, 9:56 AM UTC

reacharavindh wrote: post #67

eric4smith wrote: post #17

IMHO self hosting your database (even in the cloud) is the best way to do it. You have control over the version. You have control over features. You have control over performance. It’s tons cheaper for greater performance - especially when you go over a few hundred gigs. Yes, the hosted ones have built in replication - but my data is far too valuable to put in the hands of a third party. If they lost it - they could…

As with most things in infrastructure, it is a tradeoff to be considered properly. A startup, where you'd rather have your engineers working on adding features to your product instead of "managing and operating" a database? Have a lot of money to throw at this problem, being aware that you are paying for convenience? then using RDS and such would probably be a better choice. An established company/product with a team…

What do you mean by "managing and operating" a database?

Applying migrations? initial setup and configuration?

Re: Ask HN: Do you self-host your database?

#82

Post by herbst » Tue, Jun 29, 2021, 9:57 AM UTC

In the 6 years of running all the infrastructure for multiple thousands of daily clicks myself I only once had a currupt database that i fixed like 15 minutes after I got the bug report via email. And that's 5 years ago.

So yeah running my own postgres, one on each of my web servers actually. Mostly because it's simply sooooooo much cheaper and I hate monetary growth limitations for my projects

Re: Ask HN: Do you self-host your database?

#83

Post by marvinblum » Tue, Jun 29, 2021, 9:59 AM UTC

We're hosting Postgres on HashiCorp Nomad and ClickHouse on a separate VM for Pirsch [0]. The Postgres db is only a few kb (maybe it's a few mb now, I haven't checked in a while), as it is only used for user accounts, settings, and some configuration. It doesn't do much so it's doing OK in the cluster using a host volume on one of the machines. ClickHouse uses more storage (don't know how much right now, but it should be less than 100mb) and resources and therefore lives on its own VM.

The main reason we self-host is privacy and cost. Postgres costs almost nothing, because it's part of the cluster we require anyways (also self-hosted) and ClickHouse can be scaled as needed. Hetzner has some really cheap VM, our whole setup, including the cluster, costs about 45€ a month.

[0] https://pirsch.io/

Re: Ask HN: Do you self-host your database?

#84

Post by mekster » Tue, Jun 29, 2021, 10:10 AM UTC

geostar wrote: post #66

vbsteven wrote: post #30

Yes, I selfhost Postgres/Redis/Mongo for small projects (db + app server on the same machine or a small number of VMs). Usually in docker, sometimes systemd units. It's amazing what you can do on a single $5-50/month machine if HA isn't super important. On larger projects (typically once k8s gets involved) I'm running on a cloud provider anyway and I might as well use a hosted version like RDS for the main database.…

Do you prefer docker over systemd units?

I always wonder what the point of running a docker version of a daemon you can easily run with OS package manager, especially for those well tested packages like databases.

And then security patches are rolled out accordingly to be updated automatically by the OS service. Not sure how good with docker container is on this part.

Re: Ask HN: Do you self-host your database?

#85

Post by cakoose » Tue, Jun 29, 2021, 10:10 AM UTC

This article convinced me to go managed for Postgres: https://rbranson.medium.com/10-things-i-hate-about-postgresq... (2020)

> The good news is that the pain caused by many of the issues brought up in this post can be reduced or eliminated by using a managed database service like Heroku PostgreSQL, Compose PostgreSQL, Amazon RDS for PostgreSQL, or Google Cloud SQL for PostgreSQL. If you can use one of these services, for the love of all that is holy, please do!

Re: Ask HN: Do you self-host your database?

#86

Post by codewithcheese » Tue, Jun 29, 2021, 10:10 AM UTC

ciconia wrote: post #52

We're currently transitioning from a multi-tenant 2TB postgres DB hosted on AWS RDS to using sqlite instead, a separate database for each client. We're doing this for multiple reasons: a) As our DB grew the service became very expensive, one of the biggest items in our AWS invoice; b) Keeping the PG servers up to date is a pain, we simply don't have time for this; c) We wanted to be able to migrate to other clouds an…

Slack used to partition their databases per workspace (customer) and then moved away from it. The "Disadvantages" section lists reasons not to do that. https://slack.engineering/scaling-datastores-at-slack-with-v...

Two disadvantages that stand out for me are:

1. You must be able to scale the database up to your biggest customer and down to the your smallest this can be increasingly difficult to provision.

2. You are stuck with customers being isolated and makes it much harder to develop features that cut across a group of customers.

Re: Ask HN: Do you self-host your database?

#87

Post by onion2k » Tue, Jun 29, 2021, 10:12 AM UTC

Yes, in the sense of running a container in the cloud. The security requirements for what I work on means it's pretty much impossible to do it any other way.

Re: Ask HN: Do you self-host your database?

#88

Post by brobdingnagians » Tue, Jun 29, 2021, 10:14 AM UTC

tut-urut-utut wrote: post #77

For personal use and side projects, I always self-host. It's so much cheaper considering the tiny size. I usually start with sqlite instead of the "real" database, and in 99% of the cases it stays so. At work, I never self-host. Life is so much easier if blame for whatever unrelated reason can be outsourced to some cloud provider or internal datacenter team. Because if we self host the database, we will be responsibl…

Same for me. If I don't have full control over it, then it is better to be managed and outsource the risk. Junior programmers can't mess with the settings and it tends to just work out of the box for most things.

For my own stuff where I have full control over it, I prefer my own setup and actually setting it up the way I like it. If something isn't working then I can change it and fix it since there isn't any red tape. So the full control goes in my favor.

Re: Ask HN: Do you self-host your database?

#89

Post by YuukiRey » Tue, Jun 29, 2021, 10:16 AM UTC

I use SQLite with litestream.io/ for backups for personal hobby projects.

At work I would always go with a managed DB.

Re: Ask HN: Do you self-host your database?

#90

Post by OJFord » Tue, Jun 29, 2021, 10:17 AM UTC

codewithcheese wrote: post #86

ciconia wrote: post #52

We're currently transitioning from a multi-tenant 2TB postgres DB hosted on AWS RDS to using sqlite instead, a separate database for each client. We're doing this for multiple reasons: a) As our DB grew the service became very expensive, one of the biggest items in our AWS invoice; b) Keeping the PG servers up to date is a pain, we simply don't have time for this; c) We wanted to be able to migrate to other clouds an…

Slack used to partition their databases per workspace (customer) and then moved away from it. The "Disadvantages" section lists reasons not to do that. https://slack.engineering/scaling-datastores-at-slack-with-v... Two disadvantages that stand out for me are: 1. You must be able to scale the database up to your biggest customer and down to the your smallest this can be increasingly difficult to provision. 2. You are…

2 can be a feature too!