Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What if OpenDocument used SQLite? (2014)

sqlite.org

Post reply on HN
Print view View original

71–80 of 310 posts

Re: What if OpenDocument used SQLite? (2014)

#71

Post by scherlock »

nextaccountic wrote: post #27
p4bl0 wrote: post #6

I'm currently working on an application where I use SQLite as the file format. I want to keep a usual workflow for users where you can make edit to your document and it only changes the file when you save it. So to open a file I copy it into the :memory: database [1], then the user can do whatever manipulation they want and I can directly make the change in the database I don't need to have a model of the document ot…

This means that like a regular app, you lose data if the app crashes or there is a power loss. It's much better to save after each operation in a temporary place (probably in ~/.local/share/application/yourapp, using XDG directories), and when the user clicks save, just copy the file into the desired location. That way, if there is a power loss and you reopen the app, it opens right back where it was doing (losing ma…

If you have a db, why not just model it as unsaved data? I.e. all changes get stored to the db, but have a flag of unsaved. If you open up a file and there are unsaved changes, you can prompt the user to either make them saved or discard them.

Re: What if OpenDocument used SQLite? (2014)

#72

Post by nxobject »

Tigress8780 wrote: post #65
m4rtink wrote: post #30

Earlier quoted context omitted.

Yeah, that's why sqlite was adopted for this back then - many devices still used FAT32 on the storage volumes where tiles we often stored/cached and that had horrendous small file performance - a plain white 130 Byte PNG tile could result in 64 kB being used.

Once we had to ship millions of extremely small files to our customer, we ended up throwing them into a MongoDB and serve them with a web server. It worked great. We tried to use an image of traditional filesystems (ext4 and fat32), but with most files being under 1 KiB, it was super wasteful.

This is an extremely low quality comment, and I accept any downvotes, but I can't resist: would you say that MongoDB was web scale?

Re: What if OpenDocument used SQLite? (2014)

#74

Post by massysett »

rini17 wrote: post #11
nyanpasu64 wrote: post #4

I was optimistic that Audacity adopting SQLite would be a substantial improvement in its file saving capabilities. In practice I encountered many gotchas: - On Linux, saving into a new file onto a root-owned but world-writable NTFS mount created in /etc/fstab, fails due to permission errors or something. Saving into an existing file works as usual. - Files are modified on disk when you edit the project in the program…

Yes it should replicate the functionality user expects - save everything into temporary file and overwrite the original file only on explicit save action. As for Git, it would benefit from using text format specifically aimed for easy diffing/merging. No idea how easy the sqlite dump is in this regard.

I set up my Git to use the SQLite dump on SQLite files when using “git diff”. This at least shows me the changes row-by-row, or shows nothing if no changes.

I don’t expect to be able to merge though.

Re: What if OpenDocument used SQLite? (2014)

#75

Post by SanderNL »

nextaccountic wrote: post #27
p4bl0 wrote: post #6

I'm currently working on an application where I use SQLite as the file format. I want to keep a usual workflow for users where you can make edit to your document and it only changes the file when you save it. So to open a file I copy it into the :memory: database [1], then the user can do whatever manipulation they want and I can directly make the change in the database I don't need to have a model of the document ot…

This means that like a regular app, you lose data if the app crashes or there is a power loss. It's much better to save after each operation in a temporary place (probably in ~/.local/share/application/yourapp, using XDG directories), and when the user clicks save, just copy the file into the desired location. That way, if there is a power loss and you reopen the app, it opens right back where it was doing (losing ma…

You are right and like you explained this is trivially easily fixed by autosaving regularly.

What I have trouble imagining is people working with documents on computers for more than a few years yet somehow failing to develop the Always Save Instinct. I regularly catch myself saving unreasonably often.

Re: What if OpenDocument used SQLite? (2014)

#76

Post by remram »

p4bl0 wrote: post #6

I'm currently working on an application where I use SQLite as the file format. I want to keep a usual workflow for users where you can make edit to your document and it only changes the file when you save it. So to open a file I copy it into the :memory: database [1], then the user can do whatever manipulation they want and I can directly make the change in the database I don't need to have a model of the document ot…

Why not use a transaction?

Re: What if OpenDocument used SQLite? (2014)

#78

Post by cduzz »

Someone wrote: post #60
nextaccountic wrote: post #27

Earlier quoted context omitted.

This means that like a regular app, you lose data if the app crashes or there is a power loss. It's much better to save after each operation in a temporary place (probably in ~/.local/share/application/yourapp, using XDG directories), and when the user clicks save, just copy the file into the desired location. That way, if there is a power loss and you reopen the app, it opens right back where it was doing (losing ma…

> and when the user clicks save, just copy the file into the desired location. To be perfectly safe, you want to rename it, not copy it. If there’s a power loss during copying, you may endcup with corrupted data. Renaming is, to coin a phrase, “more atomic” than copying (on Linux, the OS says it is atomic. ISO C says it, too, but POSIX doesn’t ( https://pubs.opengroup.org/onlinepubs/000095399/functions/re... : “This…

Well, copying is simply not atomic in linux; directory entry operations (rename, link, unlink) are atomic. There is a definition somewhere that says how many bytes may be written atomically; that's it -- past that writes are not atomic.

The prior comment of "model user interactions in the database" seems spot on -- just keep track of what the user's doing as unsaved data in the database and commit it (in the appropriate way) to the DB as it happens; save is just another user action.

Presumably sqlite has figured out how to write to the filesystem without corrupting itself even in a variety of adverse scenarios?

If not, commit to a temporary copy of the DB that gets renamed to the "main" name periodically or when the app closes. There's an xzzzbit meme there somewhere, yo.

Re: What if OpenDocument used SQLite? (2014)

#79

Post by josephg »

SanderNL wrote: post #75
nextaccountic wrote: post #27

Earlier quoted context omitted.

This means that like a regular app, you lose data if the app crashes or there is a power loss. It's much better to save after each operation in a temporary place (probably in ~/.local/share/application/yourapp, using XDG directories), and when the user clicks save, just copy the file into the desired location. That way, if there is a power loss and you reopen the app, it opens right back where it was doing (losing ma…

You are right and like you explained this is trivially easily fixed by autosaving regularly. What I have trouble imagining is people working with documents on computers for more than a few years yet somehow failing to develop the Always Save Instinct. I regularly catch myself saving unreasonably often.

Users shouldn’t ever need to adapt to computer crashes like this. Software should always auto save or have recovery files or something. As a principle, software should hold anything a user inputs with reverence.

Re: What if OpenDocument used SQLite? (2014)

#80

Post by mort96 »

scherlock wrote: post #71
nextaccountic wrote: post #27

Earlier quoted context omitted.

This means that like a regular app, you lose data if the app crashes or there is a power loss. It's much better to save after each operation in a temporary place (probably in ~/.local/share/application/yourapp, using XDG directories), and when the user clicks save, just copy the file into the desired location. That way, if there is a power loss and you reopen the app, it opens right back where it was doing (losing ma…

If you have a db, why not just model it as unsaved data? I.e. all changes get stored to the db, but have a flag of unsaved. If you open up a file and there are unsaved changes, you can prompt the user to either make them saved or discard them.

That feels like it requires the data model to be very different? The file format would essentially need to be a list of changes, with a "committed" flag.

Like, if someone changes some text in a paragraph, you can't just model that as "this paragraph now contains this new text". You have to model it as "this paragraph used to contain this text, but an uncommitted change changed it to this other text". User deletes an image? You have to still store the image and all the references to it, but with an uncommitted change to delete the image and remove the references to it.

And maybe that's a good thing, maybe a git-like system where the history of every change is tracked is what you want. But it certainly doesn't feel like it'd be appropriate for every application and file format.

Post reply on HN