Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Do you self-host your database?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 236 posts

Re: Ask HN: Do you self-host your database?

#51

Post by sam_lowry_ »

buro9 wrote: post #45
devoutsalsa wrote: post #39

Earlier quoted context omitted.

I'm mostly thinking about all the stuff that goes wrong. Corrupted data files, server(s) die, monitoring, getting paged, backups, testing restores, yada yada. It works until it doesn't, and then the value you derive from self-hosting is questioned when there's a failure. If you can self-insure against shit happening & like doing so, then it's probably worth it. Mostly just point this out for the potential self-hoster…

If uptime is critical and you don't want this to be a full time job then don't do it... Opt for PaaS. But for me uptime isn't critical, I can happily be down for a few days if needed. It hasn't happened once yet but I can tolerate it were it to happen. Plus this is how I learn. By doing it I learn how to do it and what should be considered. When it goes wrong I learn how to avoid that in future and what I should do t…

I had a similar setup, only with MySQL as the database since 2003. Started on a server in a closet of my own apartment, then moved to Leaseweb, then Hetzner. The project had its highest point some 14 years ago, with well over 1mln page views per month, but I could still fit it on a single bare metal server.

As the project started to fade out, I moved to Hetzner Cloud to reduce the cost of standby infrastructure.

It was down for 4 hours in a row several times over these 18 years, but otherwise the uptime was close to six sigma (under 3 hours per year).

It's surprising how rock solid hosting providers are. Can't imagine hosting anything substantial on AWS after all these years.

Re: Ask HN: Do you self-host your database?

#52

Post by ciconia »

We're currently transitioning from a multi-tenant 2TB postgres DB hosted on AWS RDS to using sqlite instead, a separate database for each client.

We're doing this for multiple reasons: a) As our DB grew the service became very expensive, one of the biggest items in our AWS invoice; b) Keeping the PG servers up to date is a pain, we simply don't have time for this; c) We wanted to be able to migrate to other clouds and even be able to offer a self-hosted version of our platform.

Re: Ask HN: Do you self-host your database?

#53

Post by raphael_kimmig »

devoutsalsa wrote: post #39
buro9 wrote: post #34

Earlier quoted context omitted.

It was very little effort to setup. Used puppet to define everything I wanted the infra to be, and have pretty much left it be. It's only reachable from the app servers, backed up to tarsnap. It's really not a lot to learn and hasn't been a chore to operate.

I'm mostly thinking about all the stuff that goes wrong. Corrupted data files, server(s) die, monitoring, getting paged, backups, testing restores, yada yada. It works until it doesn't, and then the value you derive from self-hosting is questioned when there's a failure. If you can self-insure against shit happening & like doing so, then it's probably worth it. Mostly just point this out for the potential self-hoster…

There are definitely some things to be considered here, however I find that most people drastically overestimate the amount of work associated with hosting things.

Also they tend to underestimate the amount of work required when using managed solutions. For example, you'll certainly want to do secondary backups and test restores even for managed options.

Re: Ask HN: Do you self-host your database?

#54

Post by simonw »

ciconia wrote: post #52

We're currently transitioning from a multi-tenant 2TB postgres DB hosted on AWS RDS to using sqlite instead, a separate database for each client. We're doing this for multiple reasons: a) As our DB grew the service became very expensive, one of the biggest items in our AWS invoice; b) Keeping the PG servers up to date is a pain, we simply don't have time for this; c) We wanted to be able to migrate to other clouds an…

I'd be fascinated to hear more about this.

Where do the SQLite files live? Are you using some kind of NFS or EFS or similar for them? Sharding them across many machines?

How are you handling backups and high availability? Are you using Litestream?

How many SQLite files do your application servers have open at any one time?

Re: Ask HN: Do you self-host your database?

#56

Post by backd00r »

sumanthvepa wrote: post #11

Yes. I self-host for my infrastructure. MariaDB and PostgresSQL. I've been self-hosting MySQL and Postgress for nearly 15 years now. But it's not something I recommend my clients doing.You need to be really careful to avoid foot guns like the one that took down new blur. For example. I don't run my DBs in containers, although I run my applications within containers.

What are the downsides of running DB in containers?

Re: Ask HN: Do you self-host your database?

#57

Post by [deleted] »

dataflow wrote: post #46
ehutch79 wrote: post #42

Earlier quoted context omitted.

They're talking about running the db on a vm, setting it up and tuning it themselves. As opposed to a managed solution which is setup and tuned and hosted on a vm controlled by another company.

I'm still unclear on if the VM here is on a machine they own/on their own premises? Or is it under someone else's control? "In the cloud" sounded like the latter but it sounds like the intention was the former?

[deleted]

Re: Ask HN: Do you self-host your database?

#58

Post by simonw »

BozeWolf wrote: post #33

Right now we (team of 12, 1 devops engineer) use mongodb atlas. While mongodb/nosql itself really is not giving us any value and that I really wished that postgresql was picked for our relational data, I must admit that atlas really is a nice product. Hosting is not only about speed and price. We use atlas (=hosted mongodb), because we get that nice dashboard with all those statistics and performance tuning hints. Al…

"Question: is there a comparable service for postgresql like mongodb atlas?"

So many options. Amazon RDS and Google Cloud SQL both offer PostgreSQL. Heroku PostgreSQL has been a solid option for years. Crunchy Data's Crunchy Bridge is a recent offering with serious talent behind it.

Re: Ask HN: Do you self-host your database?

#59

Post by ies7 »

eric4smith wrote: post #17

IMHO self hosting your database (even in the cloud) is the best way to do it. You have control over the version. You have control over features. You have control over performance. It’s tons cheaper for greater performance - especially when you go over a few hundred gigs. Yes, the hosted ones have built in replication - but my data is far too valuable to put in the hands of a third party. If they lost it - they could…

In my country, after certain size, pg on aws ec2 with a few capable dba/devops salary still cheaper than rds or aurora

Re: Ask HN: Do you self-host your database?

#60

Post by weavie »

vira28 wrote: post #32
kureikain wrote: post #27

I self hosted everything from Postgres to ElasticSearch. IMHO, managed solution ins't that much reliable than running your own with the right planning. Failover strategy of AWS RDS is just absurb. It's literally just promote other instance, switch DNS over. And I do see issue where the new master are behind a few transaction compare with the replica... Managed database is just to move the responsibility when the data…

Amazing. What are some advices that you would give for someone who is looking to self host?

Never run postgres with the default configuration. It comes with some extremely conservative settings so you are unlikely to get decent performance out of the box.
Post reply on HN