Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Do you self-host your database?

news.ycombinator.com

Post reply on HN
Print view View original

231–236 of 236 posts

Re: Ask HN: Do you self-host your database?

#231

Post by mattmanser »

tpetry wrote: post #26

I do host a medium-sized PostgreSQL database, but i would love not to. I need the control you won't have with a cloud offering (you can't install extensions) but i don't want to do all the other steps, and you always have the fear that backups do not work correctly. And i am not the only one hosting it by my own, many people are using self-hosting PaaS like dokku, flynn or caprover. And all these solutions have a com…

If you genuinely have a fear about backups, why don't you just automate a weekly/daily restore of your backup and have it check the data makes sense (e.g. there's an order in the order table within the last 24 hours, that the orders have increased, etc.).

Re: Ask HN: Do you self-host your database?

#232

Post by cutthegrass2 »

exikyut wrote: post #217
cutthegrass2 wrote: post #71

Earlier quoted context omitted.

This sounds super interesting, are you able to share any further detail? I'm currently wrestling with a 35TB Aurora DB and considering options for future growth. The system is multi-tenant and one of the options on the table is breaking out the tenants and hosting them on either their own system or alongside smaller subset of tenants.

I must admit I'm very curious as to what real-world thing(s) translate to 35TB of stored data (that presumably isn't large blobs, like video or ML models). (And oh wow I just realized the billing must be amazing...)

Yep, the AWS cost footprint of this system is eye-watering. The compute spend is 2x the RDS spend!

Can't say too much, but it's Financial Services related.

Re: Ask HN: Do you self-host your database?

#233

Post by ngrilly »

mattmanser wrote: post #229
ciconia wrote: post #52

We're currently transitioning from a multi-tenant 2TB postgres DB hosted on AWS RDS to using sqlite instead, a separate database for each client. We're doing this for multiple reasons: a) As our DB grew the service became very expensive, one of the biggest items in our AWS invoice; b) Keeping the PG servers up to date is a pain, we simply don't have time for this; c) We wanted to be able to migrate to other clouds an…

You'll probably come to regret that. A DB per customer is a maintenance nightmare. I remember a discussion here a few years ago about it, and there were so many voices chiming up to say 'we did this, terrible idea'.

Why?

Re: Ask HN: Do you self-host your database?

#234

Post by exikyut »

cutthegrass2 wrote: post #232
exikyut wrote: post #217

Earlier quoted context omitted.

I must admit I'm very curious as to what real-world thing(s) translate to 35TB of stored data (that presumably isn't large blobs, like video or ML models). (And oh wow I just realized the billing must be amazing...)

Yep, the AWS cost footprint of this system is eye-watering. The compute spend is 2x the RDS spend! Can't say too much, but it's Financial Services related.

Ah, a gigantic time series event firehose. (That was what I was curious about.)

I guess this cues the stereotypical "have you looked at kdb+" question, then, which you've probably already fielded :)

(https://kx.com)

IIUC, kdb+ does in fact have a real DB hiding in it, and it's not all just "weird programming language" - but it does seem to require/prefer a reasonable amount of buy-in to the way it does things to be valuable. I guess the lateral question is whether its performance obsession extends to smaller on-disk footprint, and whether the savings would be worth throwing out all the the existing Aurora integration work. (Answer: very probably not.)

Re: Ask HN: Do you self-host your database?

#235

Post by grumpy-cowboy »

jesterson wrote: post #37
eric4smith wrote: post #17

IMHO self hosting your database (even in the cloud) is the best way to do it. You have control over the version. You have control over features. You have control over performance. It’s tons cheaper for greater performance - especially when you go over a few hundred gigs. Yes, the hosted ones have built in replication - but my data is far too valuable to put in the hands of a third party. If they lost it - they could…

and most importantly, you have control over your DATA.

This is the MOST important reason no one talk about.

Re: Ask HN: Do you self-host your database?

#236

Post by ledgerdev »

dublin wrote: post #178

Yes. A few years ago, was all-in for the cloud. Today, we have way too many examples of the cloud providers shutting down people they disagree with, for reasons that usually have nothing to do with terms of service. Given the rapidly expanding judgment of speech that is deemed "unacceptable", this is a real problem for everyone, not just conservative or free speech sites. One example of many: America's Frontline Doct…

> Completely agree. I am so disheartened with how co-workers don't seem to take any issue with the aws service suspensions. They are just like oh well, it won't ever happen to us and they must have deserved it. New rule for myself when designing any system now is no cloud vendor lock in allowed.
Post reply on HN