131–140 of 187 posts
Earlier quoted context omitted.
You don’t even need a ram disk imho, databases already cache everything in memory and only writes reach the disk. Just try and cold-start your database and run a fairly large select twice.
Also the OS will cache a lot of the reads even if your database isn’t sophisticated enough or tuned correctly. Still could be a fun exercise, as with all things on here.
Earlier quoted context omitted.
Trading systems bluntly keep everything in RAM, in preallocated structures. It all depends on the kind of tradeoffs you're willing to make.
I used to work on a telecom platform (think something that runs 4G services), where every node was just part of an in-memory database that replicated using 2PC and just did periodic snapshot to avoid losing data. Basically processes were colocated with their data in the DB.
Even if you weren't using erlang, his influence (and in general, ericsson) permeates the telecom industry.
I’m not from “start up world” but in the end, few things give me more comfort and lack of surprises down the line than just having a relational database with built in redundancy/transaction logs/back up/recovery. Sure there might always be edge cases (lack of money, regulations, specialist software offering) but in the vast majority of cases - just get a database.
With many enterprise databases these days, often "incremental" or other seemingly required backup modes are not included in the "community source" versions; perhaps because surely if you want your database to be backed up safely and then come back online safely, you certainly will fall into the "contact us for quote" enterprise customer demographic.
At least, with SQLite, copying even a hot (in-use) db file to a remote server will usually "just work", with the potential loss of a few transactions, but with most other database/servers, you definitely can't just backup the data directory occasionally and call it a day.
We didn’t want to build something complicated, so we implemented our own raft consensus layer. Have you considered just using Redis?
I once saw a project in the wild where the "database" was implemented using filesystem directories as "tables" with JSON files inside as "rows". When I asked people working on it if they considered Redis or Mongo or Postgres with jsonb columns, they just said they considered all of those things but decided to roll out their own db anyway because "they understood it better". This article gives off the same energy. I r…
This isn't innovation though. You literally just write your server like you would for a single machine, then wrap it any of the available Raft libraries. AWS and other cloud providers are money printers because a lot of engineers are insanely tied into established patterns of doing things and can't think through things at a fundamental level. Ive seen company backends where their entire AWS stacks could be replaced b…
Raft does consensus. Raft does not do persistence to disk, WAL, crash recovery, indexing, vacuuming (you're using tombstones for your deletes, right?), or any of the other necessary pieces of a database. That's not mentioning how such a system has no query engine, so every piece of data you're looking up in every place you need data is traversing your bespoke data structures.
What you described isn't a database. Keeping some disposable values cached isn't a database.
I’m not from “start up world” but in the end, few things give me more comfort and lack of surprises down the line than just having a relational database with built in redundancy/transaction logs/back up/recovery. Sure there might always be edge cases (lack of money, regulations, specialist software offering) but in the vast majority of cases - just get a database.
It's interesting you say "backup/recovery" as a strong point of relational databases (servers), because backup and recovery on hot databases have always been a challenge. With many enterprise databases these days, often "incremental" or other seemingly required backup modes are not included in the "community source" versions; perhaps because surely if you want your database to be backed up safely and then come back o…
For stuff like MariaDB a quick search also finds options to perform snapshots, backups, restores etc.
And if you need to be super high available, set up a distributed DB like Cassandra - you lose the relational and transaction part, but at least you’re running a product with known failure modes and known ways to prevent/circumvent them.
I guess my bigger point is that besides “don’t roll your own crypto”, I’d also advice not to roll your own DB. There’s a lot of known stuff in the market, all built by people who made and fixed the mistakes you’re going to make a long time ago.
This architecture is roughly how HashiCorp's Nomad, Consul, and Vault are built (I'm one of the maintainers of Nomad). While it's definitely a "weird" architecture, the developer experience is really nice once you get the hang of it. The in-memory state can be whatever you want, which means you can build up your own application-specific indexing and querying functions. You could just use sqlite with :memory: for the…