I wonder how flexible is the SQLite binary format to allow to design a tool that would take a SELF binary/db and rewrite it to make BLOB values more mmap-able (OS page aligned) in order to allow shortcuts in the self-exec loader.
You'd have to at least increase the page size to allow 4kB of consecutive data without interfering with the page header. OR you restructure the code and data to work around the page headers.
Or you could write a new SQLite VFS that somehow separates the page headers from the page data.
I can buy that an object file can be viewed as a relational db. Why SQLite though? Why not SQL query engine over the object file using a virtual table abstraction? I’m not seeing how most the SQLite features, with the exception of a subset of the query engine would translate over. If the author wants to make a case for including schema metadata in an object file, again why SQLite? This strikes me a lot as someone who…
(author)
In the post I say I did write sqlelf which is just that, sqlite + virtual tables prior to this.
As far as agent workflows are concerned whats needed is the other direction. Store the files wherever but let an agent interact with it as if its on local FS via bash or other languages using their file io.
Yeah, I have been amazed for an entire life how many tools the IT industry invented during last 50 years to just disguise a database. OK, in '70s, '80s or '90s when compute and storage resources were limited and every bit counted, specialized formats did make sense. But nowadays we'd save enormous efforts by just packaging stuff in SQLite databases. Microsoft's proprietary file formats (Office, Power BI etc.), OpenOf…
At first I thought it was still for a word document to be saved in SQL but then I thought about embedded images and it makes complete sense. Makes more sense for a Publisher/Impress app though as they have more self-contained objects. In Word/Writer the main text is one huge run of characters, how would you sqlize that effectively?
You can do any of this in-memory wife ELF of course, it's just that nobody does and nobody's written the tools for it (the most obscure tool I know of is `ld -r`, which links two or more .o files and produces a new .o file instead of an executable)
For sure, the main advantage is that since the SQLite file format is easy to use compared to ELF, it's more likely that such experiments would be even tried at all. I got another one: bundling multiple (completely separate) executables with shared dependencies in the same SQLite, selected by argv[0] when called, like what busybox does
> I realized something that bugged me. ELF is already a database. Even more broadly applied: Every base of data is already a database (that's what the compound word means). Programs like sqlite and postgres used to be called by the more precise term "Relational Data Base Management Software" or "RDBMS" until their use became so widespread that they were colloquially called databases instead. The very first data base…
I understand what you are saying, but that's not close to the original meaning of the term, or at least not what I have seen in early papers. Originally the idea was a single central repository of information which would be used to run a company: so similar to systems like SAP, but obviously a lot simpler and with the assumption of running on a mainframe. It didn't refer to a particular choice of data structure, but to the way that multiple business processes would share information.
> I realized something that bugged me. ELF is already a database. Even more broadly applied: Every base of data is already a database (that's what the compound word means). Programs like sqlite and postgres used to be called by the more precise term "Relational Data Base Management Software" or "RDBMS" until their use became so widespread that they were colloquially called databases instead. The very first data base…
Like the Unicode Character Database, which is a series of text files in similar-seeming but disparate formats (it now also has an XML representation):
The lack of mmap is a problem, especially given how dynamic link heavy Linux distros are. It’s going to cause a lot of extra RAM usage and the kernel won’t know to dump those pages efficiently and will swap them to disk. The extra RAM usage will probably be recovered - the kernel typically these days has deduplication capabilities - but the lack of efficient swap is a problem.