Live data from Hacker News

What if OpenDocument used SQLite? (2014)

sqlite.org

61–70 of 310 posts

Re: What if OpenDocument used SQLite? (2014)

#61
post #6

I'm currently working on an application where I use SQLite as the file format. I want to keep a usual workflow for users where you can make edit to your document and it only changes the file when you save it. So to open a file I copy it into the :memory: database [1], then the user can do whatever manipulation they want and I can directly make the change in the database I don't need to have a model of the document ot…

> I can directly make the change in the database I don't need to have a model of the document other than its database format.

I don't get your point. Are you saying that you don't need to have a model of the document other than the model of the document? What's the nuance I'm missing?

Re: What if OpenDocument used SQLite? (2014)

#62
post #46

Love the vibe of artivles, which present let's say reason-driven development vs habit-driven. Why habit? Well, I can imagine back at the time OpenOffice was a fresh project, it went like this: "XML is going to stay forever and everybody uses XML, so ofc we use one... oh, it is so big! And there are many files, so we just zip'em"... To be fair, the author of this excellent article doesn't even say about getting rid of…

> Well, I can imagine back at the time OpenOffice was a fresh project

OpenOffice was born when Sun bought StarOffice, which was initially released in 1985 (on Z80 and certainly without any XML). So the project itself was far from fresh. OpenDocument was developed from OpenOffice.org XML format which was developed after Sun bought StarOffice in 1999. At the time XML was not used everywhere, but it was very much in vogue, certainly at Sun where the official line was that Java (created at Sun) and XML are going to conquer the world.

Re: What if OpenDocument used SQLite? (2014)

#63
post #43

Earlier quoted context omitted.

The complaint is not “it isn’t good” but rather “it is not replaceable”. Since SQLite is so powerful, once you specify it as a format, you are stuck with SQLite forever.

Which is also "not a big issue", since it's a recommended Library of Congress storage format, and supported long term: https://www.sqlite.org/locrsf.html https://www.sqlite.org/lts.html

It is somewhat of a problem: the development team is very small, they don't take outside contributions (so nobody outside the core team really builds up expertise over time), and the vast majority of tests are proprietary. I hope they have a contingency plan just in case (some sort of a dead man's switch that publishes the test suite under a permissible license) as it would probably be quite difficult for others to maintain the same quality without those tests, or re-implement them in a reasonable time frame.

Re: What if OpenDocument used SQLite? (2014)

#64
post #46

Love the vibe of artivles, which present let's say reason-driven development vs habit-driven. Why habit? Well, I can imagine back at the time OpenOffice was a fresh project, it went like this: "XML is going to stay forever and everybody uses XML, so ofc we use one... oh, it is so big! And there are many files, so we just zip'em"... To be fair, the author of this excellent article doesn't even say about getting rid of…

Could you clarify the "XML is dead" comment? Don't all the major document formats still use zipped xml? I had to interface with an xml format recently, and that isn't something I ever did, and when I went looking for a crate that parses an xml schema I kept running across this whole xml is dead thing. But it still seems to be everywhere.

Re: What if OpenDocument used SQLite? (2014)

#65
post #30

Earlier quoted context omitted.

Many filesystems have an issue with tens of thousands or more files in a single directory, which is exactly what you can get with map tiles. No wonder sqlite is faster.

Yeah, that's why sqlite was adopted for this back then - many devices still used FAT32 on the storage volumes where tiles we often stored/cached and that had horrendous small file performance - a plain white 130 Byte PNG tile could result in 64 kB being used.

Once we had to ship millions of extremely small files to our customer, we ended up throwing them into a MongoDB and serve them with a web server. It worked great.

We tried to use an image of traditional filesystems (ext4 and fat32), but with most files being under 1 KiB, it was super wasteful.

Re: What if OpenDocument used SQLite? (2014)

#66
post #61
post #6

I'm currently working on an application where I use SQLite as the file format. I want to keep a usual workflow for users where you can make edit to your document and it only changes the file when you save it. So to open a file I copy it into the :memory: database [1], then the user can do whatever manipulation they want and I can directly make the change in the database I don't need to have a model of the document ot…

> I can directly make the change in the database I don't need to have a model of the document other than its database format. I don't get your point. Are you saying that you don't need to have a model of the document other than the model of the document? What's the nuance I'm missing?

I suppose this is in the context where you will be syncing up the changes to a backend server which will also be storing the document in an SQL database. Normally, you might expect that data format on the client to be JSON/XML/something else, and you'd need to maintain logic that marshalls the document representation

    SQL  In-memory representation  Disk format. 
With SQL on the client, in theory you only now need to maintain

    SQL  In-memory representation
Obviously I'm skirting over the format you would use to send either entire documents or partial updates of documents over the wire.

Re: What if OpenDocument used SQLite? (2014)

#67
post #49

I don't want people to read my drafts. That could be highly embarassing, and they should not make it into the final saved document. Past version and undo history should be stored separately from the document. They should be stored out of tree where they wont be commited into some git repository or be automatically synced or anything like that.

I want to be able to read my drafts, until I decide to bake a publication version.

Did you read the other part of my comment? Where I said to store the draft, but not in the document itself?

Re: What if OpenDocument used SQLite? (2014)

#68
post #14

Coupling a file format to SQLite smells wrong. SQLite is good, but it is also fairly unique in this space. Why? Because it’s hard to replicate everything it does, because it does a lot. But… for this case, do we need it do a lot? No, not really. We don’t need the full SQL standard, a query optimiser, etc etc for basic (+ safe) transaction semantics and the ability to store data in a basic table structure. Perhaps the…

I found the transactional aspect surprisingly difficult, especially with concurrent file access. SQLITE_BUSY handling was quite hard at the time. I know that serialization failures are expected in transaction processing, but for SQLite it was very difficult to tell persistent failures (say, due to self-deadlock) apart from transient concurrent update problems. For transient failure, you can re-execute the closure defining the transactional operation, but for persistent failure, that's of course pointless.

Part of the problem is that sqlite3_stmt combines aspects of both prepared statements and result sets. There is a tendency to keep them around to cache the compiled bytecode (prepared statement), but your might code might stop mid-iteration (result set), maybe holding a lock at this point. This can lead to surprising lock-upgrade failures. In the end, I wrote extensive error reporting using sqlite3_next_stmt, sqlite3_stmt_busy, sqlite3_sql, just to weed out those issues. The entire transaction retry code I wrote is full of optional logging and many comments, even though it was just for my own personal use. Before that, I wrote transaction retry logic for PostgreSQL, and that was so much easier (but it was before fully SERIALIZABLE transactions arrived).

The other surprise is that “ A transaction committed in WAL mode with synchronous=NORMAL might roll back following a power loss or system crash.” (https://sqlite.org/pragma.html#pragma_synchronous), but that wasn't relevant to my application.

Re: What if OpenDocument used SQLite? (2014)

#69

Earlier quoted context omitted.

It is not just fat32 and overhead up to cluster size; once I had 800k tiles in a single directory on ntfs. It was unusable. The only thing that I was able to do is to tar it up and move to a machine with xfs, where I was able to sort it up into more balanced subdirs and then move it back (for processing using windows-only tool). Just tarring that single directory up took several days.

It's not just ntfs. I tested this a few months ago in a pretty unscientific manner using ~50 million files in one directory. btrfs was unusable (not only that particular directory, but the whole filesystem became noticeably slower). ext4 was ok. xfs didn't break a sweat. I don't recall any practical difference when compared against a nested tree like ├── aa │ ├── aa │ │ └── aaaaf3ee5e6b4b0d3255bfef95601890afd80709 │…

[deleted]

Re: What if OpenDocument used SQLite? (2014)

#70
post #6

I'm currently working on an application where I use SQLite as the file format. I want to keep a usual workflow for users where you can make edit to your document and it only changes the file when you save it. So to open a file I copy it into the :memory: database [1], then the user can do whatever manipulation they want and I can directly make the change in the database I don't need to have a model of the document ot…

This means that like a regular app, you lose data if the app crashes or there is a power loss. It's much better to save after each operation in a temporary place (probably in ~/.local/share/application/yourapp, using XDG directories), and when the user clicks save, just copy the file into the desired location. That way, if there is a power loss and you reopen the app, it opens right back where it was doing (losing ma…

Good ole .filename.swp
Post reply on HN