Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Do you self-host your database?

news.ycombinator.com

Post reply on HN
Print view View original

111–120 of 236 posts

Re: Ask HN: Do you self-host your database?

#111

Post by panta82 »

codewithcheese wrote: post #86
ciconia wrote: post #52

We're currently transitioning from a multi-tenant 2TB postgres DB hosted on AWS RDS to using sqlite instead, a separate database for each client. We're doing this for multiple reasons: a) As our DB grew the service became very expensive, one of the biggest items in our AWS invoice; b) Keeping the PG servers up to date is a pain, we simply don't have time for this; c) We wanted to be able to migrate to other clouds an…

Slack used to partition their databases per workspace (customer) and then moved away from it. The "Disadvantages" section lists reasons not to do that. https://slack.engineering/scaling-datastores-at-slack-with-v... Two disadvantages that stand out for me are: 1. You must be able to scale the database up to your biggest customer and down to the your smallest this can be increasingly difficult to provision. 2. You are…

My experience mirrors that.

If you have a dynamic set of databases or even tables, be ready to invest into tooling to recreate features you take for granted in a normal database.

Re: Ask HN: Do you self-host your database?

#112

Post by andix »

mekster wrote: post #84
geostar wrote: post #66

Earlier quoted context omitted.

Do you prefer docker over systemd units?

I always wonder what the point of running a docker version of a daemon you can easily run with OS package manager, especially for those well tested packages like databases. And then security patches are rolled out accordingly to be updated automatically by the OS service. Not sure how good with docker container is on this part.

The great benefit of docker is, that you can replicate the exact same environment on a number of different machines. So the developer on Windows has exactly the same database version and configuration as the production server. Especially useful if you use some extensions or external modules for your database.

If you host something serious and you don't need special builds of databases it is probably better not to use docker. But it's a bit more effort.

Re: Ask HN: Do you self-host your database?

#113

Post by vbsteven »

geostar wrote: post #66
vbsteven wrote: post #30

Yes, I selfhost Postgres/Redis/Mongo for small projects (db + app server on the same machine or a small number of VMs). Usually in docker, sometimes systemd units. It's amazing what you can do on a single $5-50/month machine if HA isn't super important. On larger projects (typically once k8s gets involved) I'm running on a cloud provider anyway and I might as well use a hosted version like RDS for the main database.…

Do you prefer docker over systemd units?

It depends. Many small projects I'm involved in already have a docker-compose.yml with database, backend and other services. Combined with a private docker registry somewhere (Gitlab, ECR). If that is available I will use docker to deploy.

If that is not available, then it depends on how complex the dependency graph is. If it is simple and I can just `apt install` all dependencies then I'll wrap the install in an Ansible Playbook and use systemd. If not I create my own docker-compose.yml for deployment.

edit: when multiplexing multiple projects on the same host (happens with very small projects) I always use docker.

Re: Ask HN: Do you self-host your database?

#114

Post by ilrwbwrkhv »

escovabr wrote: post #93
ciconia wrote: post #52

We're currently transitioning from a multi-tenant 2TB postgres DB hosted on AWS RDS to using sqlite instead, a separate database for each client. We're doing this for multiple reasons: a) As our DB grew the service became very expensive, one of the biggest items in our AWS invoice; b) Keeping the PG servers up to date is a pain, we simply don't have time for this; c) We wanted to be able to migrate to other clouds an…

How do you deal with common sqlite lock file when using it for multiple users?

Not a problem up to 100,000 customers imp

Re: Ask HN: Do you self-host your database?

#115

Post by gsa »

Lunrtick wrote: post #44
mushufasa wrote: post #43

Earlier quoted context omitted.

> supported extensions in my experience, vendors support few extensions (for good reason), but lots of time you need a specific use-case in that long tail of unsupported extensions. for example, timescaledb isn't supported anywhere, so the options are managing it yourself or using their own hosted Postgres version.

DigitalOcean does offer Timescaledb in the their managed postgres offering AFAIK, along with quite a few other extensions (PostGIS etc)

The version of Timescaledb offered by DigitalOcean is Apache licensed, which is somewhat limited compared to the community licensed version.

Re: Ask HN: Do you self-host your database?

#116

Post by mosselman »

simonw wrote: post #54
ciconia wrote: post #52

We're currently transitioning from a multi-tenant 2TB postgres DB hosted on AWS RDS to using sqlite instead, a separate database for each client. We're doing this for multiple reasons: a) As our DB grew the service became very expensive, one of the biggest items in our AWS invoice; b) Keeping the PG servers up to date is a pain, we simply don't have time for this; c) We wanted to be able to migrate to other clouds an…

I'd be fascinated to hear more about this. Where do the SQLite files live? Are you using some kind of NFS or EFS or similar for them? Sharding them across many machines? How are you handling backups and high availability? Are you using Litestream? How many SQLite files do your application servers have open at any one time?

Interested in more info as well.

Also, how do you handle migrations, both schema as well as data?

Re: Ask HN: Do you self-host your database?

#117

Post by Jnr »

ilrwbwrkhv wrote: post #114
escovabr wrote: post #93

Earlier quoted context omitted.

How do you deal with common sqlite lock file when using it for multiple users?

Not a problem up to 100,000 customers imp

I used it 10 years ago and initially locks were a slight problem. Not sure if this has been solved somehow in sqlite itself.

I solved the problem by handling all the DB operations in a separate thread that handled write and read queues. It certainly was not as easy as using mysql/postgresql where server does everything for you. Nowadays there are probably plenty of libraries that handle that for you.

Re: Ask HN: Do you self-host your database?

#118

Post by BozeWolf »

simonw wrote: post #58
BozeWolf wrote: post #33

Right now we (team of 12, 1 devops engineer) use mongodb atlas. While mongodb/nosql itself really is not giving us any value and that I really wished that postgresql was picked for our relational data, I must admit that atlas really is a nice product. Hosting is not only about speed and price. We use atlas (=hosted mongodb), because we get that nice dashboard with all those statistics and performance tuning hints. Al…

"Question: is there a comparable service for postgresql like mongodb atlas?" So many options. Amazon RDS and Google Cloud SQL both offer PostgreSQL. Heroku PostgreSQL has been a solid option for years. Crunchy Data's Crunchy Bridge is a recent offering with serious talent behind it.

But i want support like atlas gives me :) amazon and google dont do that. I know i can get hosted solutions, but that really is not the same. Enterprisedb maybe?

Re: Ask HN: Do you self-host your database?

#119

Post by teddyh »

“Self-host” is such a weird word. Having your own stuff yourself should be the default, should it not? I mean, you don’t “self-drive” your car, nor “self-work” your job. The corresponding words instead exists for the opposites: You can have a chauffeur and you can outsource your job.

I think the problem is entirely caused by the US having absolutely abysmal private internet speeds and capacity. Since you can’t then have your own server at home, you are forced to have it elsewhere with sensible internet connections.

It’s as if, in an alternate reality, no private residences had parking space for cars; no garages, no street parking. Everyone would be forced to either use public transport, taxis and chauffeur services to get anywhere. Having a private vehicle would be an expensive hobby for the rich and/or enthusiasts, just like having a personal server is in our world.

— Me, 2019-10-13: https://news.ycombinator.com/item?id=21235957#21240357

Re: Ask HN: Do you self-host your database?

#120

Post by mekster »

andix wrote: post #112
mekster wrote: post #84

Earlier quoted context omitted.

I always wonder what the point of running a docker version of a daemon you can easily run with OS package manager, especially for those well tested packages like databases. And then security patches are rolled out accordingly to be updated automatically by the OS service. Not sure how good with docker container is on this part.

The great benefit of docker is, that you can replicate the exact same environment on a number of different machines. So the developer on Windows has exactly the same database version and configuration as the production server. Especially useful if you use some extensions or external modules for your database. If you host something serious and you don't need special builds of databases it is probably better not to use…

I always thought using a local machine is an anti pattern. There are plenty of virtualization software, one should just run the same OS as the production via VMware locally or give each person their own dev environment in the cloud with the same OS and be done with it than trying to align the environment partially.
Post reply on HN