Yes. Our stuff won’t fit on any cloud instances for a reasonable amount of money.
Ask HN: Do you self-host your database?
21–30 of 236 posts
Re: Ask HN: Do you self-host your database?
#22Last place I worked self-hosted. Price performance on old hardware was (and seemingly is) drastically better than cloud DB for smaller usage. Current work is using AWS RDS and it seems to perform pretty poorly unless we're willing to pay a lot more than our scale justifies. (Frankly, I suspect running our prod DB on a developer laptop would outperform the low-cost RDS setup we're using...)
Re: Ask HN: Do you self-host your database?
#23Yes. I even go so far as to embed the database in my applications.
In general, this is implemented as two main flows:
- Data collected from "the edge": Web servers that serve ads, or receive form fill-outs for lead generation, that do nothing but record this information in a logfile
- Configuration and Reporting pushed from "the hub": A central processing node (usually in a hot/warm configuration, but I've been known to use things like raft here) that receives information from flow 1, makes a decision and writes the result to flow 2.
Because the "data" I want to materialise to clients always either fits into ram, or is in a coherent stream ready to deliver to a client exactly, my applications are very fast: I can always just add another node (in the right geographic place), however I also noticed another really interesting benefit: It is much easier to develop for than using an external database.
I have a general purpose tool I wrote decades ago that moves logfiles around reliably (and at low latency), merges them together, implements failover, etc, so I really don't have very much to write to make a new application. Sometimes I experiment with the encoding of the payload, but sometimes not.
Re: Ask HN: Do you self-host your database?
#24Yes. A Postgres which is currently only 1TB in size. It's read heavy, the write workload is very low in tens to spikes of low hundreds per second whilst the read is in thousands per second. The workload is stable during traffic spikes due to app caching and CDN caching (for guests). I do this for cost reasons, self hosting the database is so much cheaper than the managed options that for the same cost I can run the e…
Cheaper in terms of $$$. If you like managing a database, I'm sure it's a good deal!
Re: Ask HN: Do you self-host your database?
#25Hmmm varies, if it's corporate it's on cloud vendors and managed, if personal or hobby projects it's unmanaged.
Re: Ask HN: Do you self-host your database?
#26I do host a medium-sized PostgreSQL database, but i would love not to. I need the control you won't have with a cloud offering (you can't install extensions) but i don't want to do all the other steps, and you always have the fear that backups do not work correctly.
And i am not the only one hosting it by my own, many people are using self-hosting PaaS like dokku, flynn or caprover. And all these solutions have a common problem, they all need to reinvent the database as a service layer. What is currently really missing is a good open-source PostgreSQL as a service appliance, something "simple" like Heroku. There have been attempts like Elephant Shed, but they all try to do too much and therefore fail in adoption as they never reach stability. Or people are forced to use complex solutions like patroni which is doing many things, but if something fails you have no clue what to do.
So what is really missing?
1. A simple old-school PostgreSQL vm image 2. Built on a copy-on-write file system to clone production environments like https://postgres.ai/ and Heroku for development reasons which will not really need any storage space. 3. A built-in backup solution like pgBackRest storing the files encrypted in some cloud storage (and restore options!) 4. It does not need a complex ui or inspection/monitoring software, there are many solutions you can run on a different machine. 5. Replication, Auto-Failover etc. are hard just make it a single server which you scale vertically, if you need horizontal scaling you have very specific needs, there can't be a one size fits all solution, so don't even try it. And a real bare-metal server with nvme disks has a lot of power it's insane how fast it is compared to cloud hosted databases.
Re: Ask HN: Do you self-host your database?
#27I self hosted everything from Postgres to ElasticSearch.
IMHO, managed solution ins't that much reliable than running your own with the right planning. Failover strategy of AWS RDS is just absurb. It's literally just promote other instance, switch DNS over. And I do see issue where the new master are behind a few transaction compare with the replica...
Managed database is just to move the responsibility when the database is down to somebody else.
Self host gives you a lot of flexibility to mix and experiment with new technologies and tooling.
Re: Ask HN: Do you self-host your database?
#28Yes. I self-host for my infrastructure. MariaDB and PostgresSQL. I've been self-hosting MySQL and Postgress for nearly 15 years now. But it's not something I recommend my clients doing.You need to be really careful to avoid foot guns like the one that took down new blur. For example. I don't run my DBs in containers, although I run my applications within containers.
Makes sense. How do you handle firewall? Just open up the source IP with the IP of wherever the Application is hosted?
Yes, limit connections in firewall. Ideally you are only listening on a private network anyway, better yet only listening on loopback or a local socket. But still firewall it. You can also configure the database to only accept connections from a particular source. Do that too.
Re: Ask HN: Do you self-host your database?
#29We self-host a postgres cluster in k8s (AKS). The main reason being that the Azure postgres offering was not fit for (our) purpose. We did start with the Azure offering, and our default is generally to use the cloud offering.
I’m also interested. We are currently using the managed pg server where we create multiple databases inside. The managed pg database does not support private endpoint if i recall and it gets pretty expensive.
Also single managed server doesn’t scale well.
Re: Ask HN: Do you self-host your database?
#30Yes, I selfhost Postgres/Redis/Mongo for small projects (db + app server on the same machine or a small number of VMs). Usually in docker, sometimes systemd units. It's amazing what you can do on a single $5-50/month machine if HA isn't super important.
On larger projects (typically once k8s gets involved) I'm running on a cloud provider anyway and I might as well use a hosted version like RDS for the main database.
It comes down to the importance/budget of the project. I'm not a Postgres expert by any means but I'm confident enough that for simple use cases I can manage selfhosted. And if I need more, hosted versions are available at a cost.
However, any hosted DB product I use has to be open source and in theory easily replaceable with a selfhosted version. After the Parse.com fiasco I'm averse to closed/proprietary components in my infrastructure.