Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What If OpenDocument Used SQLite?

sqlite.org

Post reply on HN
Print view View original

51–60 of 109 posts

Re: What If OpenDocument Used SQLite?

#51

Post by johncolanduoni »

Retr0id wrote: post #29
liuliu wrote: post #24

One thing I would call out, if you use SQLite as an application format: BLOB type is limited to 2GiB in size (int32). Depending on your use cases, that might seem high, or not. People would argue that if you store that much of binary data in a SQLite database, it is not really appropriate. But, application format usually has this requirement to bundle large binary data in one nice file, rather than many files that yo…

You can split your data up across multiple blobs

Also you almost certainly want to do this anyway so you can stream the blobs into/out of the network/filesystem, well before you have GBs in a single blob.

Re: What If OpenDocument Used SQLite?

#52

Post by wakawaka28 »

kstrauser wrote: post #46
wakawaka28 wrote: post #44

Earlier quoted context omitted.

What is it that makes you think Lunduke is pseudo-intellectual? He certainly doesn't try to pose as a scholar. If you are like most of his haters, you just refuse to believe that smart people can be conservatives.

There’s no way to discuss Lunduke without getting into politics, so I’ll leave it that Lunduke is clearly a very intelligent person who IMO mistakes his knowledge in some areas for general expertise in other unrelated fields. It’s a common trap to fall into. See also: Ben Carson. Both of them are obviously intelligent and highly skilled in their professional fields. And both have let that convince themselves that the…

I don't think Lunduke is a Ben Carson type. That would be ridiculous. He has opinions about things outside his area of expertise, like all of us, but he also has some unique experiences like having worked for Microsoft and OpenSUSE. His opinions on tech are pretty solid. I also agree with his politics for the most part.

Re: What If OpenDocument Used SQLite?

#53

Post by kstrauser »

wakawaka28 wrote: post #52
kstrauser wrote: post #46

Earlier quoted context omitted.

There’s no way to discuss Lunduke without getting into politics, so I’ll leave it that Lunduke is clearly a very intelligent person who IMO mistakes his knowledge in some areas for general expertise in other unrelated fields. It’s a common trap to fall into. See also: Ben Carson. Both of them are obviously intelligent and highly skilled in their professional fields. And both have let that convince themselves that the…

I don't think Lunduke is a Ben Carson type. That would be ridiculous. He has opinions about things outside his area of expertise, like all of us, but he also has some unique experiences like having worked for Microsoft and OpenSUSE. His opinions on tech are pretty solid. I also agree with his politics for the most part.

I would hear what he has to say about his tech experiences. I would not be in a room where he was discussing his politics.

Re: What If OpenDocument Used SQLite?

#54

Post by lifthrasiir »

hedora wrote: post #40
lifthrasiir wrote: post #36

Earlier quoted context omitted.

Yeah, but only if SQLite did support that mode in some built-in VFS implementation...

Which network filesystems are still corrupting sqlite files? Sqlite on NFSv3 has been rock solid for some NFS servers for a decade. Maybe name and shame?

Specifically I had an issue over 9p used by WSL2. (I never thought it was networked before this incident.)

Re: What If OpenDocument Used SQLite?

#55

Post by chungy »

lifthrasiir wrote: post #34

SQLite can't be reliably used in networked file systems because it heavily relies on locking to be correctly implemented. I recently had to add a check for such file systems in my application [1] because I noticed a related corruption firsthand. Simpler file formats do not demand such requirements. SQLite is certainly good, but not for this use. [1] https://github.com/lifthrasiir/angel/commit/50a15e703ef2c1af...

That's pretty broad and over-generalized. Networking file systems without good lock support is almost always a bad setup by an administrator. Both NFS and CIFS can work with network-wide locks just fine.

SQLite advises against using a networking file system to avoid potential issues, but you can successfully do it.

Re: What If OpenDocument Used SQLite?

#56

Post by wzdd »

chris_wot wrote: post #38
agwa wrote: post #28

If you're going to use SQLite as an application file format, you should: 1. Enable the secure_delete pragma https://antonz.org/sqlite-secure-delete/ > so that when your user deletes something, the data is actually erased. Otherwise, when a user shares one of your application's files with someone else, the recipient could recover information that the sender thought they had deleted. 2. Enable the options described at…

"Most applications can use SQLite without having to worry about bugs in obscure SQL inputs." And then they recommend SQLite as a document interchange format.

Although this is indeed a worrying statement, it seems true to me. Most users of sqlite control the SQL they use. The problem I would expect from using a database document interchange format is that a maliciously crafted database could result in a CVE. The page acknowledges this possibility, even while pointing out (in their CVE list) that it hasn't happened so far, or is rare (it's hard to parse some of their descriptions).

Re: What If OpenDocument Used SQLite?

#57

Post by cpach »

afiori wrote: post #45
lifthrasiir wrote: post #34

SQLite can't be reliably used in networked file systems because it heavily relies on locking to be correctly implemented. I recently had to add a check for such file systems in my application [1] because I noticed a related corruption firsthand. Simpler file formats do not demand such requirements. SQLite is certainly good, but not for this use. [1] https://github.com/lifthrasiir/angel/commit/50a15e703ef2c1af...

In that case the application would keep a temporary file and copy over when saving

Maybe, but how would the application know if /data/foo.bar is a local file or mounted via NFS/SMB/etc?

Re: What If OpenDocument Used SQLite?

#58

Post by jdboyd »

chungy wrote: post #55
lifthrasiir wrote: post #34

SQLite can't be reliably used in networked file systems because it heavily relies on locking to be correctly implemented. I recently had to add a check for such file systems in my application [1] because I noticed a related corruption firsthand. Simpler file formats do not demand such requirements. SQLite is certainly good, but not for this use. [1] https://github.com/lifthrasiir/angel/commit/50a15e703ef2c1af...

That's pretty broad and over-generalized. Networking file systems without good lock support is almost always a bad setup by an administrator. Both NFS and CIFS can work with network-wide locks just fine. SQLite advises against using a networking file system to avoid potential issues, but you can successfully do it.

Are the typical Synology, Qnap, or TrueNAS devices with default Linux, macOS and Windows clients going to be set up correctly by default? If any of the typical things someone is likely to setup following wizards in a home or small office is likely to result in lock not working correctly for SQLite, then it is fair for them to warn against using it on a network file system.

As an application format, you don't generally expect people to be editing an ODF file at the same time though, so network locking doesn't really disqualify it for use as a document format.

Re: What If OpenDocument Used SQLite?

#59

Post by kvdveer »

lifthrasiir wrote: post #34

SQLite can't be reliably used in networked file systems because it heavily relies on locking to be correctly implemented. I recently had to add a check for such file systems in my application [1] because I noticed a related corruption firsthand. Simpler file formats do not demand such requirements. SQLite is certainly good, but not for this use. [1] https://github.com/lifthrasiir/angel/commit/50a15e703ef2c1af...

In the context of this article, that's largely irrelevant: ZIP cannot be used in a multi-user scenario at all, so even if sqlite isn't perfect, it's still miles better than the ZIP format it replaces in this thought experiment.

Re: What If OpenDocument Used SQLite?

#60

Post by jdboyd »

floating-io wrote: post #3

An interesting skim, but it would have been more meaningful if it had tackled text documents or spreadsheets to show what additional functionality would be enabled with those beyond "versioning". Maybe it's just me, but I see the presentation functionality as one of the less used aspects of the OpenOffice family.

What he listed as the first improvement, "Replace ZIP with SQLite" would certainly apply to the other ODF formats.

He advocates breaking the XML into smaller pieces in SQLite. I suppose making each slide a new XML record could make sense. Moving over to spreadsheets, I don't know how ODF does it now, but making each sheet a separate XML could make sense.

Thinking about Write documents, I wonder what a good smaller unit would be. I think one XML per page would be too fine a granularity. You could consider one record per chapter. I doubt one record per paragraph would make sense, but it could be fun to try different ideas.

Post reply on HN