Live data from Hacker News

Ask HN: Do you self-host your database?

news.ycombinator.com

41–50 of 236 posts

Re: Ask HN: Do you self-host your database?

#41
post #16

Earlier quoted context omitted.

Makes sense. How do you handle firewall? Just open up the source IP with the IP of wherever the Application is hosted?

Yes, limit connections in firewall. Ideally you are only listening on a private network anyway, better yet only listening on loopback or a local socket. But still firewall it. You can also configure the database to only accept connections from a particular source. Do that too.

Thanks for clarifying.

> you are only listening on a private network anyway what if i host my app in say some where else than the DB?

> better yet only listening on loopback or a local socket. For that app and db needs to run on the same server. Isn't it?

Re: Ask HN: Do you self-host your database?

#42

IMHO self hosting your database (even in the cloud) is the best way to do it. You have control over the version. You have control over features. You have control over performance. It’s tons cheaper for greater performance - especially when you go over a few hundred gigs. Yes, the hosted ones have built in replication - but my data is far too valuable to put in the hands of a third party. If they lost it - they could…

> self hosting your database (even in the cloud) > my data is far too valuable to put in the hands of a third party Confused, you self-host "in the cloud" without a third party involved? What does that mean? Are you not counting the party hosting your machine?

They're talking about running the db on a vm, setting it up and tuning it themselves. As opposed to a managed solution which is setup and tuned and hosted on a vm controlled by another company.

Re: Ask HN: Do you self-host your database?

#43
post #38
post #26

I do host a medium-sized PostgreSQL database, but i would love not to. I need the control you won't have with a cloud offering (you can't install extensions) but i don't want to do all the other steps, and you always have the fear that backups do not work correctly. And i am not the only one hosting it by my own, many people are using self-hosting PaaS like dokku, flynn or caprover. And all these solutions have a com…

> I need the control you won't have with a cloud offering (you can't install extensions) Depends on who you go with. Google's managed postgres thing lets you install supported extensions. https://cloud.google.com/sql/docs/postgres/extensions

> supported extensions in my experience, vendors support few extensions (for good reason), but lots of time you need a specific use-case in that long tail of unsupported extensions.

for example, timescaledb isn't supported anywhere, so the options are managing it yourself or using their own hosted Postgres version.

Re: Ask HN: Do you self-host your database?

#44
post #38

Earlier quoted context omitted.

> I need the control you won't have with a cloud offering (you can't install extensions) Depends on who you go with. Google's managed postgres thing lets you install supported extensions. https://cloud.google.com/sql/docs/postgres/extensions

> supported extensions in my experience, vendors support few extensions (for good reason), but lots of time you need a specific use-case in that long tail of unsupported extensions. for example, timescaledb isn't supported anywhere, so the options are managing it yourself or using their own hosted Postgres version.

DigitalOcean does offer Timescaledb in the their managed postgres offering AFAIK, along with quite a few other extensions (PostGIS etc)

Re: Ask HN: Do you self-host your database?

#45
post #34

Earlier quoted context omitted.

It was very little effort to setup. Used puppet to define everything I wanted the infra to be, and have pretty much left it be. It's only reachable from the app servers, backed up to tarsnap. It's really not a lot to learn and hasn't been a chore to operate.

I'm mostly thinking about all the stuff that goes wrong. Corrupted data files, server(s) die, monitoring, getting paged, backups, testing restores, yada yada. It works until it doesn't, and then the value you derive from self-hosting is questioned when there's a failure. If you can self-insure against shit happening & like doing so, then it's probably worth it. Mostly just point this out for the potential self-hoster…

If uptime is critical and you don't want this to be a full time job then don't do it... Opt for PaaS.

But for me uptime isn't critical, I can happily be down for a few days if needed. It hasn't happened once yet but I can tolerate it were it to happen.

Plus this is how I learn. By doing it I learn how to do it and what should be considered. When it goes wrong I learn how to avoid that in future and what I should do to minimise risk of it recurring.

I find the database to be the easiest past to manage, it's file storage and availability of that which I outsource to AWS S3. The volume of that is harder for me to trivially solve. Self hosting Ceph seems more intimidating. Postgres is easy though.

Re: Ask HN: Do you self-host your database?

#46

Earlier quoted context omitted.

> self hosting your database (even in the cloud) > my data is far too valuable to put in the hands of a third party Confused, you self-host "in the cloud" without a third party involved? What does that mean? Are you not counting the party hosting your machine?

They're talking about running the db on a vm, setting it up and tuning it themselves. As opposed to a managed solution which is setup and tuned and hosted on a vm controlled by another company.

I'm still unclear on if the VM here is on a machine they own/on their own premises? Or is it under someone else's control? "In the cloud" sounded like the latter but it sounds like the intention was the former?

Re: Ask HN: Do you self-host your database?

#47
post #23

Yes. I even go so far as to embed the database in my applications. In general, this is implemented as two main flows: - Data collected from "the edge": Web servers that serve ads, or receive form fill-outs for lead generation, that do nothing but record this information in a logfile - Configuration and Reporting pushed from "the hub": A central processing node (usually in a hot/warm configuration, but I've been known…

So you're using web apps that run sqlite? With some logic on a central server, which then updates all the web apps?

No. I mean, I have used sqlite with dbmail (not a web app) in this way sometime late 2000s, but if you're just going to load the table into memory anyway, you might as well skip the SQL and just record the data.

For example, I did a "sandwich store" app that would list orders in JSON. These were written to a log file via a PHP script. The replication tool would copy them to the "home base" (the store itself). When the kitchen would print an order, it would record a log line saying the order was started, and this would be replicated via a different logfile to the two web servers. The user might submit an order and not see it right away, but I hid this using a cookie so you might only notice if you were using two web browsers logged in with the same user. The receiver on the edge would write out files for each order containing the most recently loaded status, so collecting the status just involved asking the servers for the contents of this file. When the order was scheduled to be delivered, another log line would update those files. And so on. Nothing really resembling a "database" here at all -- just files and memory.

Most recently, my ad server (erlang) has a ets table (cached in dets to speed recovery) that contains all of the publisher+targeting details as a key, and the list of matching advertisers. This table gets updated when home-base records a configuration change (say, because an operator adds a new site or advertiser). Configuration changes look something like this:

     {{market,[{id,>}]},
      {{2021,6,28},{10,40,9}},
      >,'nodename',
      {patch,market,
          [{patch,demands,
               [{patch,demand,
                    [{id,>}],
                    [{patch,customers,
                         [{patch,market_customer,
                              [{customer,[...]}],
                              [{patch,...},{...}]}]}]}]}]}},
     {{usr,[{id,>}]},
      {{2021,6,28},{9,52,21}},
      >,'nodename',
      {patch,usr,[{patch,accts,[{insert,>}]}]}},
     {{market,[{id,>}]},
      {{2021,6,28},{9,50,42}},
      >,'nodename',
      {patch,market,[]}},
     {{market,[{id,>}]},
      {{2021,6,28},{9,49,37}},
      >,'nodename',
      {patch,market,
          [{patch,demands,
               [{patch,demand,
                    [{id,>}],
                    [{patch,customers,
                         [{delete,market_customer,[...]},
                          {insert,market_customer,...}]}]}]}]}},
They're actually stored as binary terms in a disk_log. A process reads the disk_log and materialises the in-memory configuration for the web services and rebuilds the ets table. Erlang has a lot of tools that (perhaps unobviously) make this very easy.

And so on.

Re: Ask HN: Do you self-host your database?

#48
We don't, the maintenance cost in man hours was way to high and replacing it with AWS Aurora made it at least as reliable with a lot less overhead.

On the plus side we can do ad hoc tests and experiments by creating a new Aurora cluster with a recent snapshot and try things out.

Re: Ask HN: Do you self-host your database?

#49
post #38
post #26

I do host a medium-sized PostgreSQL database, but i would love not to. I need the control you won't have with a cloud offering (you can't install extensions) but i don't want to do all the other steps, and you always have the fear that backups do not work correctly. And i am not the only one hosting it by my own, many people are using self-hosting PaaS like dokku, flynn or caprover. And all these solutions have a com…

> I need the control you won't have with a cloud offering (you can't install extensions) Depends on who you go with. Google's managed postgres thing lets you install supported extensions. https://cloud.google.com/sql/docs/postgres/extensions

They don't let you "install" an extension, the `CREATE EXTENSION` command is just enabling an installed extension for a specific database. There are a ton more extensions than Google Cloud SQL or AWS RDS support. Both of them for example don't have the often used hypopg extensions which allows you to hypothetical create an index to check whether it's used for your queries and see how your execution plan changes. Building indexes for large databases can take a lot of time.

Re: Ask HN: Do you self-host your database?

#50

Earlier quoted context omitted.

> supported extensions in my experience, vendors support few extensions (for good reason), but lots of time you need a specific use-case in that long tail of unsupported extensions. for example, timescaledb isn't supported anywhere, so the options are managing it yourself or using their own hosted Postgres version.

DigitalOcean does offer Timescaledb in the their managed postgres offering AFAIK, along with quite a few other extensions (PostGIS etc)

But does it offer the new TimescaleDB 2.0 with all the new features? They changed their license and made all enterprise features free but only for their Community License (for self-hosting). Cloud providers are only allowed to use the fully open source Apache licensed extensions which is missing some parts.
Post reply on HN