171–180 of 236 posts
I used WordPress as my datastore for a while on personal projects. I could wrestle it mostly into shape and use the built-in rest api. WP is my day job so I know it well and like it, but I'm not a backender, so it was a bit clunky and time consuming to get things the way I wanted. Then I needed to host that WP instance somewhere. That was also a pain. I decided last year to try and use Node/Express/Mongo for my backe…
Earlier quoted context omitted.
> Just "have no idea, datacenter team is working on it. You can ask them?" has done wonders for my mental health and job satisfaction Never seems to have much benefit for me, because I'm typically the one who answers the phone/email/etc, and "the datacenter team" never does. No one is happy with that, and I still get the calls/email/messages, demanding the 'fix' or ETA. I would rather things be more under my control,…
But the thing is in most of the companies you don't have full control over the whole stack. Even if you have "full control" over the database, you don't have control over networking, firewall, OS, "security" patching, VMs, Docker, Kubernetes, Load balancers, vendors managing parts of the infra, internet provider, hosting provider ... Not even datacenter team may have control over all of it, but at least that's their…
You seem to have exceptional customers. Like the parent comment, my customers wouldn't have that. I'm their point of contact, and they want me to do the legwork and provide them with updates. They want to complain to me directly, often citing that they have enough problems with their business to care about problems my business is facing. We'd be out of business if we insisted they talk to our vendors directly. They would straight up laugh in our faces as they fired us, and frankly, I'm with them, since I wouldn't put up with that from our vendors either.
For self-hosting, MariaDB (or Percona or MySQL) with InnoDB is really unbeatable. - Version upgrades don't require migrations. - Replication is pretty easy, well understood and allows version differences between the two ends. Nowadays they even got crash resilience right. - It doesn't require VACUUM or any other regular maintenance. - XtraBackup is awesome. I've been running a pretty large and rather critical MariaDB…
When I last tried, it, AWS RDS still required EBS-backed disks. It may be acceptable for 99% of use cases, but that means there's a ceiling around 60k IOPS, even with PIOPS disks. Outside of RDS, you're afforded a bit more creativity in your performance profile: multiple disks in RAID 0, tiered storage (burstable AND, rather than OR, provisioned IOPS storage), and instance store, to name a few. I actually once witnes…
I use a small library that encapsulates the use-case (including some schema migration stuff that should probably be replaced with flyway).
https://github.com/mschaef/sql-file
There's a lot that can be said about this approach, both for and against, but I find that it brings a lot of power, is easy to set up, and generally stays out of the way. Given that the fundamental abstraction presented to user code is essentially SQL with connection pooling, there are also good escape strategies to get to architectures that would support however much capacity I'm ever likely to need.
Earlier quoted context omitted.
It was very little effort to setup. Used puppet to define everything I wanted the infra to be, and have pretty much left it be. It's only reachable from the app servers, backed up to tarsnap. It's really not a lot to learn and hasn't been a chore to operate.
I'm mostly thinking about all the stuff that goes wrong. Corrupted data files, server(s) die, monitoring, getting paged, backups, testing restores, yada yada. It works until it doesn't, and then the value you derive from self-hosting is questioned when there's a failure. If you can self-insure against shit happening & like doing so, then it's probably worth it. Mostly just point this out for the potential self-hoster…
Someone else managing that DB doesn't prevent me (or someone malicious) from destroying the data, various infra screwups still are a thing (though perhaps less common). The only difference is if doing the same things through the managed DB admin panel takes meaningfully less effort than ssh on a self-hosted box; and it seems a wash - in the managed DB is easier to use complicated features than when self-hosted, but they tend to be more complex for simple things.
One example of many: America's Frontline Doctors, famous for their dissemination of factual information about Hydroxychloroquine and Ivermectin for Covid treatment, had their hosting pulled by AWS just a few weeks back, despite the ever-growing evidence that their stance is scientifically supported. (And I don't care if they were lying their butts off, AWS still shouldn't have pulled their hosting.)
Cloud services offer awesome leverage, which is a huge advantage, but anyone relying on them for their business or organization to operate is being completely irresponsible.
When I last tried, it, AWS RDS still required EBS-backed disks. It may be acceptable for 99% of use cases, but that means there's a ceiling around 60k IOPS, even with PIOPS disks. Outside of RDS, you're afforded a bit more creativity in your performance profile: multiple disks in RAID 0, tiered storage (burstable AND, rather than OR, provisioned IOPS storage), and instance store, to name a few. I actually once witnes…
Beyond having an IOPS ceiling, the other problem is latency spikes caused by EBS failures. At my last job where we had a write heavy RDS instance, an EBS failure would occur almost monthly, with no warning, and cause write latency to spike 3-5x for a couple minutes. This was in 2017-2018.