Live data from Hacker News

SQLite as an Application File Format

sqlite.org

61–70 of 108 posts

Re: SQLite as an Application File Format

#61
post #9

Somehow my first thought from the title was using sqlite as a format for applications. So like a replacement for ELF. I think this idea is both fascinating and horrifying.

Forget elf, imagine having a SQLite file that stores elf, exe and DMG binaries. I would not mind working on something like this.

Not that at all, but interesting in its own right - https://pypi.org/project/sqlelf/ explore ELF via SQL.

Re: SQLite as an Application File Format

#62

Most application's file formats are structured as a tree, not as flat tables. If your application's data is flat tables or name-value pairs then SQLite is an obvious choice. But if it is tree structured then it is less obvious. You can still save your tree in JSON format as a blob in a SQLite table but in this case the benefits are fewer. But if in addition to the JSON you have images or other binary data then once a…

I am not really classically trained on the subject but I think this is the idea behind relational storage, it is to have better extraction options, you don't have to treat your data as a single document at a time.

Naively, most data looks hierarchical and the instinctive reaction is to make your file format match. But if you think of this as a set of documents stacked on top of each other if you take the data as a bunch of 90 degree slices down through the stack now your data is relational, you loose the nice native hierarchical format, but you gain all sorts of interesting analysis and extraction options.

It is too bad relational data types tend to be so poorly represented in our programming languages, generally everything has to be mapped back to a hierarchical type.

Re: SQLite as an Application File Format

#65

Something to consider when using SQLite as a file format is compression (correct me if I'm wrong!). You might end up with a large file unless you consider this, and can't/won't just gz the entire db. Nothing is compressed by default.

It can be compressed, see https://sqlite.org/sqlar.html

Please do not use second resolution mtime (cannot represent the high accuracy mtime that modern OSs use, so packing and unpacking , or causes differences eg in rsync), or build anything new using DEFLATE (it is slow and cannot really be made fast).

Re: SQLite as an Application File Format

#66
post #30

Earlier quoted context omitted.

It makes it easy for me to find additional commentary for things I’m interested in, and I appreciate that. There’s no policing going on. If you find it off-putting, that’s your problem.

as i said and you ignored, all it does is put people off from having fresh conversations.

You do realize that dang himself frequently aggregates related discussions, and thanks people for doing so.

And that the previous discussions of the same URL are readily available at the top of the topic, via the "past" link.

So either HN itself is actively discouraging discussions, which seems unlikely, or your perception of this is askew.

Re: SQLite as an Application File Format

#67

The Acorn macOS app uses SQLite in a similar way: https://flyingmeat.com/acorn/docs/technotes/ACTN002.html

Recently reverse engineered the Money Pro backup format, it's a binary file with SQLite with some additional XML information backed in. It feels like they're purposefully making it harder for users to export their data in a useful format, especially after the changes they made to their financial model.

Re: SQLite as an Application File Format

#68
post #33
post #9

Somehow my first thought from the title was using sqlite as a format for applications. So like a replacement for ELF. I think this idea is both fascinating and horrifying.

Or a replacement for Access

https://docs.devart.com/odbc/sqlite/access.htm

https://github.com/sam-ludlow/access-linker

Re: SQLite as an Application File Format

#69
post #30

Earlier quoted context omitted.

It makes it easy for me to find additional commentary for things I’m interested in, and I appreciate that. There’s no policing going on. If you find it off-putting, that’s your problem.

as i said and you ignored, all it does is put people off from having fresh conversations.

I directly addressed that in my last sentence.

Re: SQLite as an Application File Format

#70

The Acorn macOS app uses SQLite in a similar way: https://flyingmeat.com/acorn/docs/technotes/ACTN002.html

Yes! Gus (the developer) also has made & maintained FMDB for many years, a nice Cocoa wrapper for the SQLite bindings.

https://github.com/ccgus/fmdb

Post reply on HN