Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What if OpenDocument used SQLite? (2014)

sqlite.org

Post reply on HN
Print view View original

101–110 of 310 posts

Re: What if OpenDocument used SQLite? (2014)

#101

Post by isoprophlex »

regularfry wrote: post #56
isoprophlex wrote: post #50

Man do I love SQLite. Over the past 1.5 yrs I've build a computer vision tool from recording hardware/software, to derp learning pipelines, to front-end; we had some requirements on the recording side that were difficult to solve with existing solutions (storing exactly timestamped camera frames, gps data, car telemetry and other metadata). Using a SQLite-backed data format for the video recordings made implementing…

> derp learning pipelines This accurately describes the majority of my efforts, too.

Honest to god this was an unintentional typo, but I decided to leave it in as it was just too juicy

Re: What if OpenDocument used SQLite? (2014)

#102

Post by eviks »

constantly wrote: post #9
vmfunction wrote: post #5

At this point, why are we still using JSON/XML when there is SQLite for new projects? Stop the non sense of JSON/XML. SQLite is like json, but very queryable. Just send SQLite files around. MongoDB also saves document db type of store space just FYI.

Any text editor in the world, even the ones that ship with the most barebones shells, can open json and xml and present their data to the user. SQLite files require opening in a DB terminal or using special software to even get to the point where one can see what’s there at all. Further the entire internet basically natively supports XML and JSON.

Outside of simple cases xml is too verbose and ugly (and in these cases usually zipped), so it's not suitable for a poor human with a plain text editor, so that doesn't give you much of a leg.

(Json has a higher threshold of complexity before it succumbs)

Re: What if OpenDocument used SQLite? (2014)

#103

Post by quickthrower2 »

eastern wrote: post #38
quickthrower2 wrote: post #23

Earlier quoted context omitted.

With JSON/XML the app owner decides the schema of the saved file, as they should. One day Sqlite will do some perfectly fine change that’ll break people who outsource their file format to it. Own your file format! That said there is some nuance and it depends what the user expects. Is you app more of an MSWord where people expect a format that is decades backward compatible and only changes on explicit save, or is it…

In sqlite the on-disk file format does not matter. All that matters is that you should be able to issue sql to the sqlite embedded library and get back the results. Freeing you from the overhead of owning (thus inventing and then maintaining) your own file format is almost the entire point of using sqlite in this manner.

It matters for 2 reasons. One, the expectation that the file changes only when you click Save is broken (as mentioned in another comment), and Two, unless you pin the version of sqlite forever then the file format may have braking changes or your need to deal with migrations.

Re: What if OpenDocument used SQLite? (2014)

#104

Post by iggldiggl »

dsego wrote: post #89
SanderNL wrote: post #75

Earlier quoted context omitted.

You are right and like you explained this is trivially easily fixed by autosaving regularly. What I have trouble imagining is people working with documents on computers for more than a few years yet somehow failing to develop the Always Save Instinct. I regularly catch myself saving unreasonably often.

That may have been true years ago in win 95 or xp days. The modern paradigm starting with google docs is that things are automatically saved and even always sharable through the cloud, making manual saving actually an atavistic leftover of a bygone era.

What if I actually don't want any changes saved because I've only opened a document for reference purposes?

Re: What if OpenDocument used SQLite? (2014)

#105

Post by knome »

rakoo wrote: post #86
p4bl0 wrote: post #6

I'm currently working on an application where I use SQLite as the file format. I want to keep a usual workflow for users where you can make edit to your document and it only changes the file when you save it. So to open a file I copy it into the :memory: database [1], then the user can do whatever manipulation they want and I can directly make the change in the database I don't need to have a model of the document ot…

Why do you use a secondary, volatile database ? Performance-wise you won't gain a lot more (we're talking about a user editing a file, so not even 1 write per second). A proposal: write directly, and automatically in the database. No more Save button. There are multiple advantages: - the system is crash-resistant. I like taking the approach of CouchDB where the only correct way to close the system is to crash it. Tha…

a save button is still good, as it allows you to keep specific checkpoints.

but the save button could simply tag specific save points in a larger table.

if the format can roll up changes to compress them, they also indicate where which variants need to be kept indefinitely.

Re: What if OpenDocument used SQLite? (2014)

#106

Post by raxxorraxor »

bawolff wrote: post #32
rini17 wrote: post #11

Earlier quoted context omitted.

Yes it should replicate the functionality user expects - save everything into temporary file and overwrite the original file only on explicit save action. As for Git, it would benefit from using text format specifically aimed for easy diffing/merging. No idea how easy the sqlite dump is in this regard.

> Yes it should replicate the functionality user expects Do users really expect this now a days? Most users use cloud apps, and almost all of those save after every operation automatically.

Which is a compromise for using browsers really. It isn't a good solution and no user really understands this and I believe it is the most hated feature of the new cloud world. Yes, leaving the page open for multiple hours might now allow you to save because your access token expired. No, communication in the background is unreliable too. Autosave is a bad band aid for a bad solution.

Doing periodic and automatic saves is good. Doing so on a document "in production" is majorly stupid. Not that I want to accidentally validate the busy work dev ops puts us through.

Re: What if OpenDocument used SQLite? (2014)

#107

Post by torstenvl »

rewmie wrote: post #61
p4bl0 wrote: post #6

I'm currently working on an application where I use SQLite as the file format. I want to keep a usual workflow for users where you can make edit to your document and it only changes the file when you save it. So to open a file I copy it into the :memory: database [1], then the user can do whatever manipulation they want and I can directly make the change in the database I don't need to have a model of the document ot…

> I can directly make the change in the database I don't need to have a model of the document other than its database format. I don't get your point. Are you saying that you don't need to have a model of the document other than the model of the document? What's the nuance I'm missing?

An in-memory data model often differs from the serialized data as it exists on disk. For example, emacs uses a gap buffer for text files; but it outputs plain linear text to disk.

Programmers often have to make software design decisions around how to represent a file in memory in order to manipulate it. For example, if I'm writing an HTML editor, should I mostly treat it like a text file (maybe a gap buffer) with syntax highlighting and auto indentation as an afterthought? Or should I maybe load the whole thing into a tree? What are the robustness and performance characteristics of each?

The commenter above was saying that using SQLite made that decision easy. He could keep traditional (or "atavistic" per the commenter upthread, depending on your perspective) load/save semantics while also making the data model easy to work with.

Re: What if OpenDocument used SQLite? (2014)

#108

Post by ReactiveJelly »

iggldiggl wrote: post #104
dsego wrote: post #89

Earlier quoted context omitted.

That may have been true years ago in win 95 or xp days. The modern paradigm starting with google docs is that things are automatically saved and even always sharable through the cloud, making manual saving actually an atavistic leftover of a bygone era.

What if I actually don't want any changes saved because I've only opened a document for reference purposes?

Have a "Read-only" checkbox. For the love of God, have a "read-only" checkbox.

Re: What if OpenDocument used SQLite? (2014)

#109

Post by cm2187 »

Etheryte wrote: post #98
cm2187 wrote: post #85

Sqlite format is smaller than the original format only because xml is super verbose, so any uncompressed binary format ends up being less than lightly zipped xml. But sqlite files aren't small. One thing I don't understand is why they don't do string deduplication in sqlite (as in you only store a string once and every other occurence is just a pointer to that string). It seems such an obvious and easy way to reduce…

My first guess is that if you always store the full string you don't need to scan the database to see if you already have the same string. Essentially you choose to use more space but reduce load. Regardless of whether you do the string deduping on inserts or async later on, you have to do it at some point and the unpredictable performance overhead might be undesirable.

Well it should be a dictionary lookup, it should be pretty fast and predictable. And for freeing it up, it should be a good candidate for reference counting.

Re: What if OpenDocument used SQLite? (2014)

#110

Post by HelloNurse »

fweimer wrote: post #68
orf wrote: post #14

Coupling a file format to SQLite smells wrong. SQLite is good, but it is also fairly unique in this space. Why? Because it’s hard to replicate everything it does, because it does a lot. But… for this case, do we need it do a lot? No, not really. We don’t need the full SQL standard, a query optimiser, etc etc for basic (+ safe) transaction semantics and the ability to store data in a basic table structure. Perhaps the…

I found the transactional aspect surprisingly difficult, especially with concurrent file access. SQLITE_BUSY handling was quite hard at the time. I know that serialization failures are expected in transaction processing, but for SQLite it was very difficult to tell persistent failures (say, due to self-deadlock) apart from transient concurrent update problems. For transient failure, you can re-execute the closure def…

If you retry your write several times and it doesn't succeed you can tell the user it is a persistent failure without agonizing too much over the diagnosis: it is persistent enough to be a significant problem, even without proof that it is an application bug.

Who would attempt to make concurrent writes to an application document format? And how wouldn't such an attempt be a user mistake? Failing to write is the solution, not the problem.

Post reply on HN