Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What if OpenDocument used SQLite? (2014)

sqlite.org

Post reply on HN
Print view View original

81–90 of 310 posts

Re: What if OpenDocument used SQLite? (2014)

#81

Post by nurbl »

orf wrote: post #45
jve wrote: post #29

Earlier quoted context omitted.

- Why not? https://www.sqlite.org/appfileformat.html - Its size is less than a megabyte: https://sqlite.org/footprint.html - 750KB if all features are enabled: https://www.sqlite.org/about.html - Looks like fair amount of functionality can be left out when compiling sqlite and with options to influence/strip down query planner: https://www.sqlite.org/compile.html - And "SQLite does not compete with client/server data…

> In the end, you don't need a database, but a library that gives you database API and behavior. Why do you need a single library that gives you a database API and behaviour? Wouldn't it be better to decouple those: provide an open, standard format that enables compact, fast, structured storage that is built to allow transaction/atomic updates. If that exists then you can plug sqlite on top of that, or something else…

A great thing about just using sqlite as the format is that you get lots of potential features. Sure most applications don't need full SQL power just to save and load data. But then at some point you might want more advanced functionality, or migrate to a new structure. And both you and your users get tools for free, e.g. to extract data or fix problems, or just look around. Other applications can quite easily read your files, without you needing to write various language libraries. Very few projects get around to building that kind of tools for their made up format.

I could agree about the single implementation, but if the alternative is making something new up I am not sure in what way that would be better.

Re: What if OpenDocument used SQLite? (2014)

#82

Post by layer8 »

nuc1e0n wrote: post #28

It's somewhat off topic I know, but is there something like sqlite but tailored for hierachical data? Like a xml document store rather than for relational data like sqlite is.

There’s ASN.1 for hierarchical data with a schema. It doesn’t provide a query language though.

Re: What if OpenDocument used SQLite? (2014)

#83

Post by chrisshroba »

SanderNL wrote: post #75
nextaccountic wrote: post #27

Earlier quoted context omitted.

This means that like a regular app, you lose data if the app crashes or there is a power loss. It's much better to save after each operation in a temporary place (probably in ~/.local/share/application/yourapp, using XDG directories), and when the user clicks save, just copy the file into the desired location. That way, if there is a power loss and you reopen the app, it opens right back where it was doing (losing ma…

You are right and like you explained this is trivially easily fixed by autosaving regularly. What I have trouble imagining is people working with documents on computers for more than a few years yet somehow failing to develop the Always Save Instinct. I regularly catch myself saving unreasonably often.

I used to have that instinct but lost it in the age of auto save. The applications (web or native) I use most often all do it for me: Google docs, Dropbox paper, notion, vscode. I don’t think I’m alone in this!

Re: What if OpenDocument used SQLite? (2014)

#84

Post by SanderNL »

josephg wrote: post #79
SanderNL wrote: post #75

Earlier quoted context omitted.

You are right and like you explained this is trivially easily fixed by autosaving regularly. What I have trouble imagining is people working with documents on computers for more than a few years yet somehow failing to develop the Always Save Instinct. I regularly catch myself saving unreasonably often.

Users shouldn’t ever need to adapt to computer crashes like this. Software should always auto save or have recovery files or something. As a principle, software should hold anything a user inputs with reverence.

I agree. Maybe as a dev I've become cynical and don't trust anything. Least of all some app holding my document.

Makes me think of the "Voting software" xkcd: https://xkcd.com/2030

"I don't quite know how to put this, but our entire field is bad at what we do, and if you rely on us, everyone will die."

Re: What if OpenDocument used SQLite? (2014)

#85

Post by cm2187 »

Sqlite format is smaller than the original format only because xml is super verbose, so any uncompressed binary format ends up being less than lightly zipped xml.

But sqlite files aren't small. One thing I don't understand is why they don't do string deduplication in sqlite (as in you only store a string once and every other occurence is just a pointer to that string). It seems such an obvious and easy way to reduce file size, memory consumption and therefore increase performance (less I/O). Is there a technical reason why this would not be desirable?

Re: What if OpenDocument used SQLite? (2014)

#86

Post by rakoo »

p4bl0 wrote: post #6

I'm currently working on an application where I use SQLite as the file format. I want to keep a usual workflow for users where you can make edit to your document and it only changes the file when you save it. So to open a file I copy it into the :memory: database [1], then the user can do whatever manipulation they want and I can directly make the change in the database I don't need to have a model of the document ot…

Why do you use a secondary, volatile database ? Performance-wise you won't gain a lot more (we're talking about a user editing a file, so not even 1 write per second).

A proposal: write directly, and automatically in the database. No more Save button. There are multiple advantages:

- the system is crash-resistant. I like taking the approach of CouchDB where the only correct way to close the system is to crash it. That way a crash is an expected situation that you actually account for, not a special case that you might forget

- there is only one database. Less code, fewer bugs.

- it is safe. A write to SQLite works or doesn't work, there is no in-between. As said in the VACUUM doc you point to: "However, if the VACUUM INTO command is interrupted by an unplanned shutdown or power lose, then the generated output database might be incomplete and corrupt"

- it is how SQLite was intended to work. And because of that, you won't have to think about it for the lifetime of SQLite

Re: What if OpenDocument used SQLite? (2014)

#87

Post by amiga386 »

orf wrote: post #14

Coupling a file format to SQLite smells wrong. SQLite is good, but it is also fairly unique in this space. Why? Because it’s hard to replicate everything it does, because it does a lot. But… for this case, do we need it do a lot? No, not really. We don’t need the full SQL standard, a query optimiser, etc etc for basic (+ safe) transaction semantics and the ability to store data in a basic table structure. Perhaps the…

Exactly. Some formats are designed, first and foremost, for interchange. SQLite is pitching that you, as an "app" owner, force the SQLite format upon your users to make it a de-facto standard, without putting the work in to make it a de-jure standard.

Show me a formalised ISO / IEC / ANSI / ETSI SQLite standard that the Richard Hipp and his company never deviates from, and the full legal search to ensure there are no patents that might affect it, and show me the multiple compatible implementations of SQLite that _all_ have these touted advantages, and _then_ we can talk about prosletizing it as a file format. If they don't, they're saying "take a hard dependency on a single-source implementation, and make all your users take it too".

XML is a formal standard. ASN.1 is a formal standard. JFIF is a formal standard. Even ZIP is a formal standard (adopted as part of standardising OpenDocument: ISO/IEC 21320-1:2015)

The most important thing about a document is that everyone _else_ can read it. Saving time on writing updates to disk is an irrelevant sideshow. Did we learn nothing from Microsoft perverting the standards bodies to try and keep its lock-in?

https://arstechnica.com/uncategorized/2008/10/norwegian-stan...

> A letter of resignation written by the departing members and made public by The Inquirer accuses the standards body of folding to pressure from Microsoft, violating its own procedural rules, and ignoring the analysis of the technical committee tasked with evaluating OOXML.

Re: What if OpenDocument used SQLite? (2014)

#88

Post by sgarland »

justinclift wrote: post #51
anonzzzies wrote: post #15

Earlier quoted context omitted.

My wife uses Audacity all day and every few days there is a corrupt sqlite file (duplicate key) which cannot be (as far as we know) repaired/reimported etc from Audacity. I can fix it manually if it's important, but usually just throw the file away and things work again.

Duplicate keys in a SQLite file sounds like an audacity bug. :(

Maybe. The SQLite list of gotchas [0] is quite something. NULLs in the PK? Sure. FKs don’t actually do anything unless you pass a PRAGMA? Why not? Etc. I could easily see someone not fully grasping just how much SQLite lets pass by default, and thus not having a test catch it.

[0]: https://www.sqlite.org/quirks.html

Re: What if OpenDocument used SQLite? (2014)

#89

Post by dsego »

SanderNL wrote: post #75
nextaccountic wrote: post #27

Earlier quoted context omitted.

This means that like a regular app, you lose data if the app crashes or there is a power loss. It's much better to save after each operation in a temporary place (probably in ~/.local/share/application/yourapp, using XDG directories), and when the user clicks save, just copy the file into the desired location. That way, if there is a power loss and you reopen the app, it opens right back where it was doing (losing ma…

You are right and like you explained this is trivially easily fixed by autosaving regularly. What I have trouble imagining is people working with documents on computers for more than a few years yet somehow failing to develop the Always Save Instinct. I regularly catch myself saving unreasonably often.

That may have been true years ago in win 95 or xp days. The modern paradigm starting with google docs is that things are automatically saved and even always sharable through the cloud, making manual saving actually an atavistic leftover of a bygone era.

Re: What if OpenDocument used SQLite? (2014)

#90

Post by colonwqbang »

orf wrote: post #45
jve wrote: post #29

Earlier quoted context omitted.

- Why not? https://www.sqlite.org/appfileformat.html - Its size is less than a megabyte: https://sqlite.org/footprint.html - 750KB if all features are enabled: https://www.sqlite.org/about.html - Looks like fair amount of functionality can be left out when compiling sqlite and with options to influence/strip down query planner: https://www.sqlite.org/compile.html - And "SQLite does not compete with client/server data…

> In the end, you don't need a database, but a library that gives you database API and behavior. Why do you need a single library that gives you a database API and behaviour? Wouldn't it be better to decouple those: provide an open, standard format that enables compact, fast, structured storage that is built to allow transaction/atomic updates. If that exists then you can plug sqlite on top of that, or something else…

The Sqlite format is open and the spec is here: https://www.sqlite.org/fileformat2.html

I haven't studied the spec in detail but it seems comprehensive.

The fact that there also exists a high-quality, stable, public domain reference implementation can't really be counted against the format, can it?

Post reply on HN