Database Review 2021

>> agreed, running pg locally is a pain

If you have a single process accessing the database, use SQLite. If not, use Postgres.

> Pretty much one of the classic desktop support calls for me was "my access database on the shared drive is corrupt"

Yes, that one >12 year old access database with two concurrent users brakes biweekly. Everything else: No problems.

Just this Thursday I "moved" a program from an old PC to a new one. We more or less support this client since 7 years. The PC is from before that. Got the call. First time hearing about that lab software. Right click on the desktop icon. Oh, it just points to an UNC path on the "main server" (Linux Samba share). Took a look into the settings. It has a path to a SQLite DB. Also just an UNC path. Only two clients using it. It worked unattended for years. Since it's stored on the "main server" it's also covered by backup. No client side installation necessary. Copy the link. Update our documentation that we know it exists, what is it used for and how it works. Done.

I see it time and time again: Desktop software that runs on one PC, produces/saves tiny amounts of data (like a temperature monitor for one fridge), but installs a SQL server. A random technician shows up, installs it and leaves. We don't know that it exists. It's therefore not part of a backup concept. It's usually only discovered when it explodes or when the PC is renewed (or when a second PostgreSQL install creates a TCP port conflict). I have the feeling only 1 out of 10 "softwares" even have a proper concept for export and import of data.

A client with like 7 or 8 workers called my because he wanted me to move his new time tracking software. The technician who also installed the NFC reader apparently installed it on the wrong PC. It took over 3 hours! It downloaded GBs and GBs of .NET, SQL Express and what not. I didn't know that going in/clicking the setup.exe. I had to call support to get the Express DB over. I can still hear the hard drive screaming. ...man... for stuff a Z80 and a CSV file would be enough. ...And I am the bad one for explaining to the client that the DB needs to be backuped. I sound like I want to upsell him something.

I know, I know. This rant is not about databases. It's about the state of IT and arrogant developers who don't live in the trenches. The "pulling down additional gigabytes of images and runtime to run the database" > "Yes. It works great." triggered me. I'm sorry. You know. Not everything is a unicorn webapp.

> Any smb client access to shared data more complicated than documents and spreadsheets just makes me twitch these days.

I know of multiple large install bases of an old medical software. It's still actively developed/supported but it is so old that it doesn't use a SQL but ISAM database. It just lies there, on a file share. No active server component to speak of. No client site installation necessary.

> What's wrong with sql express? Assuming you fit within its size constraints?

If it is not used for something a CSV file would be enough. If the costumer is told that it needs to be installed "on a server". If we get involved from the beginning so we know what is what. If the costumer doesn't expect me to be able to move it out of the blue within seconds. Than sql express is fine. ...which reminds me of that one time, when that well-hung developer of a software for the energy sector didn't knew that express only can allocate up to 2GB. Obviously he was blamed us for delivering a faulty server...

CockroachDB and Yugabyte are both wire-compatible with Postgres and follow its syntax and feature set to a large extent. They offer a reasonably clean upgrade path to a distributed database while still using the same tools you use with vanilla Postgres.

Seeing more and more projects that are either compatible or are a value-add on top of Postgres. Not seeing the same level variability of ecosystem in the MySQL world. There's MariaDB, which adds cool features like temporal tables, but PG has the aforementioned distributed DBs along with Citius, TimescaleDB for time series, AWS Redshift for OLAP, ZomboDB for enhanced full text search, Apache MADLib for big data machine learning within the database, etc.

I 100% agree that most projects don't need anything beyond SQLite (or its OLAP cousin DuckDB). More and more it looks like a relatively clean migration path from the small to the heavily concurrent to the high end niche. Great time to be a data worker.

Dynamodb is very scalable and effective at keeping operational cost down. It succeeds at it's goals by trading off on lots of things though. As a developer it can feel almost hostile. Adhoc queries are limited almost forcing you to have a complete second copy of the data. Access patterns must be designed up front and hope you don't need to change much when you hit scale. The tooling is meh. It's easier to make a scalable database when you push all those problems on your devs. Constraints are tricky. Transactions must be carefully designed. It's nice to know that almost definitely read and write throughput will scale and you can reduce the DBA/DevOps staff because AWS handles it.