Relational binary data serialization.
Cap'n Proto is a great piece of engineering that got quite a lot of things right. (For those who aren't familiar, Cap'n Proto is a data serialization format that is able to be very fast by encoding data in the same way modern processors encode data normally - with fixed width data types and pointers.) I think it would be nice to have a data serialization protocol that uses the same general concepts, but addresses a couple limitations in that format, including:
- Cap'n Proto doesn't allow you to edit messages in a robust way. You can't change the size of a list/string in a message, you can't replace one object with another without leaking "garbage", etc. The data model is simply not designed with fast editing in mind.
- The data model is document based so it is inconvenient or impossible to capture certain kinds of relationships with a schema.
My idea is to address these limitations using the battle tested relational model and the massive amount of knowledge that's been accumulated about how to efficiently implement relational systems. In this serialization format, a message would be a veritable relational dataset, complete with a schema and multiple tables. Messages would be organized into pages, each page representing a node in a b tree, as in a normal rdbms. You can add, edit, and delete rows as necessary, and just send the binary encoding of the database over the wire directly. The utility of this system is obvious: a client could, for example, read an entire database from one server, add a row to a table (without parsing the rest of the message, which are in other tables on other pages), and forward the new database directly to another server. Being able to quickly edit even large datasets in this way would be a huge boon.