Git for Data – A TerminusDB Technical Paper [pdf]

TerrminusDB originated in 2015 when Kevin Feeney and myself started working on the information architecture for the Seshat Global Historical Databank, an ambitious project to store information about every society in human history. We needed a database that could enable collaboration among a very distributed team on a shared database whose primary function was the curation of high quality datasets with a very rich structure, storing information about everything from religious practices to geographic extent. We trialled a number of graph databases and RDBMSs but found all of them wanting (though for different reasons). Thus we began our journey creating our own database from scratch. In designing the database, we focused on the elements we felt were weakest in existing information architectures, namely the ability to have very rich schemas and very fine grained revision control which was data aware - enabling features such as branch, merge, push, pull, fork etc. Essentially, we wanted to bring the sort of automation present in code pipelines using git with CI/CD to the realm of databases. With the latest release incorporating our own terminus-store backend, we now have a database architecture that is a substantial way along the journey to providing this new approach to data curation and management.

Awesomness

Thanks - we thought that too! I might re post to see if we just got timing wrong.

I haven't seen the OCaml Irmin before - very interesting and some parallel ideas. I think this sort of store is going to become a popular choice as it hits some pain points around versioning and data pipe lines. I don't know if you've had a chance to read the paper, but the data CI/CD type workflows is really what we are thinking about.