Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Do you self-host your database?

news.ycombinator.com

Post reply on HN
Print view View original

161–170 of 236 posts

Re: Ask HN: Do you self-host your database?

#161

Post by Slartie »

jrochkind1 wrote: post #139
ciconia wrote: post #52

We're currently transitioning from a multi-tenant 2TB postgres DB hosted on AWS RDS to using sqlite instead, a separate database for each client. We're doing this for multiple reasons: a) As our DB grew the service became very expensive, one of the biggest items in our AWS invoice; b) Keeping the PG servers up to date is a pain, we simply don't have time for this; c) We wanted to be able to migrate to other clouds an…

> Keeping the PG servers up to date is a pain, we simply don't have time for this I've never used AWS RDS (or similar), but I had been thinking that part of the advantage of RDS was you didn't need to "keep the PG server up to date". But apparently I was wrong! What does this look like on RDS?

You mostly need to do that if you have a security division that's looking for reasons to justify their existence. One reason involving little actual work on their own side is to constantly nag other teams to install updates "for security reasons", regardless of how much this actually contributes to security. Which usually isn't that much in case of RDS not exposed to the public network at all.

Re: Ask HN: Do you self-host your database?

#162

Post by simonw »

BozeWolf wrote: post #118
simonw wrote: post #58

Earlier quoted context omitted.

"Question: is there a comparable service for postgresql like mongodb atlas?" So many options. Amazon RDS and Google Cloud SQL both offer PostgreSQL. Heroku PostgreSQL has been a solid option for years. Crunchy Data's Crunchy Bridge is a recent offering with serious talent behind it.

But i want support like atlas gives me :) amazon and google dont do that. I know i can get hosted solutions, but that really is not the same. Enterprisedb maybe?

Crunchy Bridge might be what you're after then.

Re: Ask HN: Do you self-host your database?

#163

Post by howaboutnope »

The idea of "outsourcing blame" mentioned in several comments seems really weird to me: If I make the choice to outsource something, and whoever I outsourced it to fucks up, I'm still the one who made the decision to outsource it. The same goes for outsourcing to someone who does a better job than I could have: then that was a great idea, and yay for me.

Re: Ask HN: Do you self-host your database?

#164

Post by ahachete »

weavie wrote: post #76
lkuty wrote: post #72

Earlier quoted context omitted.

Could you elaborate on this or point us to documentation (url, book, ...) about it ?

The postgres docs are excellent, https://www.postgresql.org/docs/13/runtime-config.html I have seen tutorials where folks go over the most essential settings, but I can't seem to find the ones I have used now, so me just doing a search for 'postgres tuning' is no better than you doing the search yourself. Maybe someone else can chime in with a good guide?

We created https://postgresqlco.nf which not only offers detailed help about all configuration parameters, but also provides a "repository" for you to upload/download your Postgres configurations, and a Tuning Guide. Hope it helps.

Re: Ask HN: Do you self-host your database?

#165

Post by benbjohnson »

Jnr wrote: post #117
ilrwbwrkhv wrote: post #114

Earlier quoted context omitted.

Not a problem up to 100,000 customers imp

I used it 10 years ago and initially locks were a slight problem. Not sure if this has been solved somehow in sqlite itself. I solved the problem by handling all the DB operations in a separate thread that handled write and read queues. It certainly was not as easy as using mysql/postgresql where server does everything for you. Nowadays there are probably plenty of libraries that handle that for you.

WAL mode has been added in the last 10 years which handles read concurrency really well IMHO. You’re still constrained to a single writer but that’s not usually a problem as long as you keep your write transactions short.

Re: Ask HN: Do you self-host your database?

#166

Post by tyingq »

simonw wrote: post #54
ciconia wrote: post #52

We're currently transitioning from a multi-tenant 2TB postgres DB hosted on AWS RDS to using sqlite instead, a separate database for each client. We're doing this for multiple reasons: a) As our DB grew the service became very expensive, one of the biggest items in our AWS invoice; b) Keeping the PG servers up to date is a pain, we simply don't have time for this; c) We wanted to be able to migrate to other clouds an…

I'd be fascinated to hear more about this. Where do the SQLite files live? Are you using some kind of NFS or EFS or similar for them? Sharding them across many machines? How are you handling backups and high availability? Are you using Litestream? How many SQLite files do your application servers have open at any one time?

SQLite mentions NFS as problematic because SQLite makes a lot of use of advisory locking. Which is supposed to work on NFS, but apparently is often buggy.

Re: Ask HN: Do you self-host your database?

#167

Post by aynyc »

cutthegrass2 wrote: post #71
ciconia wrote: post #52

We're currently transitioning from a multi-tenant 2TB postgres DB hosted on AWS RDS to using sqlite instead, a separate database for each client. We're doing this for multiple reasons: a) As our DB grew the service became very expensive, one of the biggest items in our AWS invoice; b) Keeping the PG servers up to date is a pain, we simply don't have time for this; c) We wanted to be able to migrate to other clouds an…

This sounds super interesting, are you able to share any further detail? I'm currently wrestling with a 35TB Aurora DB and considering options for future growth. The system is multi-tenant and one of the options on the table is breaking out the tenants and hosting them on either their own system or alongside smaller subset of tenants.

Done that before, where my largest clients are exponentially bigger than my smallest client. The key is to realize one-size-fits-all doesn't work, not in the long term. I'm talking about feature development, storage, etc.. In the end, we break out large clients onto their own instances of RDS, where smaller clients stay multi-tenants. They share the same schemas, etc.. We wrote a lot of utility scripts to migrate clients from small to big, and vice versa. Smallest clients get automatic minor and major upgrades to latest features almost immediate. Larger clients generally only get major upgrades.

Re: Ask HN: Do you self-host your database?

#168

Post by kurko »

simonw wrote: post #54
ciconia wrote: post #52

We're currently transitioning from a multi-tenant 2TB postgres DB hosted on AWS RDS to using sqlite instead, a separate database for each client. We're doing this for multiple reasons: a) As our DB grew the service became very expensive, one of the biggest items in our AWS invoice; b) Keeping the PG servers up to date is a pain, we simply don't have time for this; c) We wanted to be able to migrate to other clouds an…

I'd be fascinated to hear more about this. Where do the SQLite files live? Are you using some kind of NFS or EFS or similar for them? Sharding them across many machines? How are you handling backups and high availability? Are you using Litestream? How many SQLite files do your application servers have open at any one time?

Also super interested, specially handling migrations and structure/schema syncs.
Post reply on HN