Live data from Hacker News

SQLite as an Application File Format

sqlite.org

101–108 of 108 posts

Re: SQLite as an Application File Format

#101
post #99

Earlier quoted context omitted.

You do realize that dang himself frequently aggregates related discussions, and thanks people for doing so. And that the previous discussions of the same URL are readily available at the top of the topic, via the "past" link. So either HN itself is actively discouraging discussions, which seems unlikely, or your perception of this is askew.

[flagged]

Fair enough, I apologize.

Re: SQLite as an Application File Format

#102

Most application's file formats are structured as a tree, not as flat tables. If your application's data is flat tables or name-value pairs then SQLite is an obvious choice. But if it is tree structured then it is less obvious. You can still save your tree in JSON format as a blob in a SQLite table but in this case the benefits are fewer. But if in addition to the JSON you have images or other binary data then once a…

Maybe not as obvious for those without formal education in """database normalization""" but it's pretty trivial to convert from a tree structure to a flat table structure using foreign key relations. Recursive queries aren't even that difficult in SQLite, so self-referential data can be represented cleanly too, if not a bit more difficult to write. IME most applications "tree structures" aren't self-referential and a…

> There's always the lazy approach of storing JSON blobs in TEXT fields, but I personally shy away from that because you lose out on a huge part of the benefits of using a SQL DB in the first place, most importantly migrations and querying/indexing.

SQLite at least provides functions to make the “querying” part of that straightforward: https://sqlite.org/json1.html

Re: SQLite as an Application File Format

#103
post #9

Somehow my first thought from the title was using sqlite as a format for applications. So like a replacement for ELF. I think this idea is both fascinating and horrifying.

I've been pondering something similar as a modern approach to fat binaries, basically around a table like

    CREATE TABLE functions (name TEXT, arch TEXT, body BLOB);
The advantage would be that binaries could be partially fattened, i.e. every function would have at least one implementation in some cross-platform bytecode (like WASM), and then some functions would get compiled to machine code as necessary, and then the really-performance-dependent functions would have extra rows for different combinations of CPU extensions or compiler optimization levels or whatever — and you could store all of these in the same executable instead of having a bunch of executables for each target.

As a bonus, it'd be possible to embed functions' source code into the executable directly this way, whether for development purposes (kinda like how things are sometimes done in the old-school Smalltalk and Lisp worlds) or for debugging purposes (e.g. when printing stack traces).

Re: SQLite as an Application File Format

#104

Earlier quoted context omitted.

> I don't understand why you'd ever want to use a relational database for that. It's a completely different paradigm. Well, it might be a relation DB or else a zipfile. Why couldn't I encapsulate a file tree in a single file ? Maybe it's tens of thousands of quite small files.

You can put tens of thousands of files in a single file lots of ways that are expressly designed for that. You don't need SQLite for that. So why would you want to use SQLite for that is my question? Mounting a database or a table as a filesystem doesn't make much sense to me. There's a very poor fit between the two paradigms. What does a subdirectory mean in a database? What does a foreign key or set of columns mean…

Maybe you misunderstand the scenario. An SQLite DB can have records where each record contains a path. This column can be used to emulate a hierarchical tree-type filesystem. There's a few different ways to represent the path information, and the parent-child connectivity among records.

Re: SQLite as an Application File Format

#105

Earlier quoted context omitted.

You can put tens of thousands of files in a single file lots of ways that are expressly designed for that. You don't need SQLite for that. So why would you want to use SQLite for that is my question? Mounting a database or a table as a filesystem doesn't make much sense to me. There's a very poor fit between the two paradigms. What does a subdirectory mean in a database? What does a foreign key or set of columns mean…

Maybe you misunderstand the scenario. An SQLite DB can have records where each record contains a path. This column can be used to emulate a hierarchical tree-type filesystem. There's a few different ways to represent the path information, and the parent-child connectivity among records.

Ok. Again, why? Why would you want to use a relational database as a filesystem rather than a file format explicitly designed for that, for mounting?

Re: SQLite as an Application File Format

#106

Earlier quoted context omitted.

Maybe you misunderstand the scenario. An SQLite DB can have records where each record contains a path. This column can be used to emulate a hierarchical tree-type filesystem. There's a few different ways to represent the path information, and the parent-child connectivity among records.

Ok. Again, why ? Why would you want to use a relational database as a filesystem rather than a file format explicitly designed for that, for mounting?

If you mean (for example) a zipfile, AFAICT there's not a whole lot of difference between them when used in this capacity.

Re: SQLite as an Application File Format

#107

Earlier quoted context omitted.

Ok. Again, why ? Why would you want to use a relational database as a filesystem rather than a file format explicitly designed for that, for mounting?

If you mean (for example) a zipfile, AFAICT there's not a whole lot of difference between them when used in this capacity.

No, you asked about mounting specifically. And I replied:

> On a Mac, you'd e.g. use and mount a disk image if you wanted to create a filesystem inside of a file. Windows has virtual hard drives, and you can do that kind of thing on Linux too.

So why wouldn't you use one of these if you need mounting? They're literally made for it.

I continue to not understand why you would want to mount a SQLite database instead of using one of these.

Re: SQLite as an Application File Format

#108

Earlier quoted context omitted.

There's a laundry list of benefits that all add up, not like one specific killer feature. Some applications really do have very complex configuration needs, but it's sorta situation dependent on whether embedding a scripting language or a database is the right solution (for really simple cases I'm more likely to reach for TOML). An incomplete list of benefits of using SQLite: - Runtime config changes for free - type…

Type safety as a benefit of SQLite? For me type safety is a negative of SQLite. Being able to store a different type that what the column is declared to store is a bug (not a feature). I also find the lack of DATE and DATETIME/TIMESTAMP to be less than ideal.

We are talking about an application file format, so "type errors" are about who's right: the application (even better, multiple equally right implementation of a specification) or random hackers altering the file in incorrect ways.

Loose type checks, e.g. NOT NULL columns of "usually" text, are loose only compared to typical SQL table definitions; compared to the leap forward of using abstract tables and changing them with abstract SQL instead of using text or byte buffers and making arbitrary changes, enforcing data types on columns would be only a marginal improvement.

Post reply on HN