Amazon Quantum Ledger Database

There are two parts of every PII storing system. The actual PII store which is super small, "mutable" with your terminology, locked down so nobody can access it without raising an alarm and usually not accessed at all except for some very limited use cases, including GDPR ones. The rest of the store just uses references to the entities sitting in the GDPR store, like a numeric id (foreign key in SQL terminology). This way any data store, SQL, datalake, etc. can be easily GDPR compliant without needing to delete data in the large data stores and this also increases security because in case of a security breach to the data stores the GDPR data cannot be accessed.

Interesting, I actually implemented a simple immutable ledger on top of mysql this year to track balance mutations of our InToken coin. The coin exists on Stellar as well but we track federated stellar accounts in our database for most of our users. This means that we represent them with a single stellar account. We allow users that clear our aml procedure to send/receive tokens to stellar.

Having our own ledger means we can cheaply do internal transfers, micro rewards, and other incentives. Doing the bookkeeping correctly and in a tamper proof way is of course important for us, which is why we built our own ledger database to ensure we don't end up with corrupted data (either through bugs or malicious activity).

I use content hashes as the id that include the id of the previous ledger entry and the key data stored in each row. The core design is pretty easy (it's basically a linked list) but the devil is in the details with this stuff since you indeed need to worry about auditing and making sure you don't end up losing transactions.

Additional headaches include dealing with concurrent transactions and the fact that mysql does not do serializable transactions (at least not in sane way). Each row should only have 1 successor meaning that every new row involves looking up the previous row, and using it's id as a parent id. So we have a select and an insert happening in a transaction. We have a simple db constraint enforcing the parent is referred only once. We retry transactions when this constraint gets violated. This does actually happen when two concurrent transactions decide to use the same parent id. If transactions were serializable, this would not happen and the second transaction would end up using the id of the first.

Another of the gotchas is that data migrations are kind of hard/impossible in an immutable data store. The only way to do it would be to effectively recreate a new database with new content hashes. So there are some things that I'd like to change that I can't actually change because it would break the content hashes. But by and large, this design is working quite well for us so far.

So, in short, it's not rocket science but hard enough that having a well supported product that does this is worth having. We're actually considering open sourcing it at some point since it seems there are quite many projects out there that use a ledger primarily to have some tamper resistant immutable and auditable log of transactions.

So you’re saying blockchain won’t help too much?

The promise and all the overhead of real distributed blockchains is that you should avoid any need to trust anyone, and if you're really paranoid then you should not trust even your own database copy. All this could be hacked just as Amazon can. Power of real distributed ledger is that as long as less than 50% of the data copies (consensus) are not hacked, then database as whole can be trusted. Now if you can hack also consensus, then even blockchain is hackable. With the "centralized ledger" you don't have the consensus protection level, therefore I see only marginal (if any) security improvement on top of just plain database with properly applied access control and auditing. You still have way more parties you need to blindly trust.

The biggest problem I see with it is putting all your eggs in one basket. One company could easily become untrustworthy without you realizing it. A lot of people trusted Enron before the scandal came out.

That problem could potentially be mitigated by doing best-of-three or best-of-five with multiple trusted parties.