Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What If OpenDocument Used SQLite?

sqlite.org

Post reply on HN
Print view View original

61–70 of 109 posts

Re: What If OpenDocument Used SQLite?

#61

Post by jdboyd »

thayne wrote: post #41
jll29 wrote: post #31

I love SQLite. As a document _exchange_/_interchange_ format, what I prefer for durability is a non-binary format (e.g. XML based). For local use, I agree SQLite might be much faster than ZIP, and of course the ability to query based on SQL has its own flexibility merits.

XML isn't great for exchange/interchange either due to security problems and inconsistencies in implementations. A big part of the problem is that xml has a lot of complexity, which leads to a bigger attack surface when parsing and processing untrusted data. And then xml entities are just inherently insecure, unless you disable some of their capabilities (like using remote files, and unlimited recursion). That said,…

Part of the problem though with saying SQLite instead of XML is a lot of things would lend themselves to XML in SQLite.

Re: What If OpenDocument Used SQLite?

#62

Post by Ekaros »

jdboyd wrote: post #61
thayne wrote: post #41

Earlier quoted context omitted.

XML isn't great for exchange/interchange either due to security problems and inconsistencies in implementations. A big part of the problem is that xml has a lot of complexity, which leads to a bigger attack surface when parsing and processing untrusted data. And then xml entities are just inherently insecure, unless you disable some of their capabilities (like using remote files, and unlimited recursion). That said,…

Part of the problem though with saying SQLite instead of XML is a lot of things would lend themselves to XML in SQLite.

Complex features are inherently complex. Say you want external resources or some scripts in document. No matter what storage format you use those are more surfaces. Problem is not storage, but what is done with information. And very often that is a lot and poorly thought out and even more poorly implemented.

Re: What If OpenDocument Used SQLite?

#63

Post by thayne »

jdboyd wrote: post #61
thayne wrote: post #41

Earlier quoted context omitted.

XML isn't great for exchange/interchange either due to security problems and inconsistencies in implementations. A big part of the problem is that xml has a lot of complexity, which leads to a bigger attack surface when parsing and processing untrusted data. And then xml entities are just inherently insecure, unless you disable some of their capabilities (like using remote files, and unlimited recursion). That said,…

Part of the problem though with saying SQLite instead of XML is a lot of things would lend themselves to XML in SQLite.

Oh, I'm not saying sqlite is better than xml for data exchange. As mentioned in other comments, sqlite's security posture towards an untrusted database is problematic. My point is that xml has problems too.

Re: What If OpenDocument Used SQLite?

#64

Post by bob1029 »

> it is still bothersome that changing a single character in a 50 megabyte presentation causes one to burn through 50 megabytes of the finite write life on the SSD.

I used to worry a lot about this but it has never once actually come up for me. 50 megabytes is a pretty extreme example, but even so if you edit this document fewer than several million times it won't matter.

Serializing the object graph all over again can be way faster than mapping into a tabular model. There are JSON serializers that can push multiple gigabytes per second per core. It might even be the case that, once you factor in the SSD controller quirks, the tabular updates could cause more blocks to be written than just dumping a big fat json stream all at once.

Re: What If OpenDocument Used SQLite?

#65

Post by mschuster91 »

jdboyd wrote: post #58
chungy wrote: post #55

Earlier quoted context omitted.

That's pretty broad and over-generalized. Networking file systems without good lock support is almost always a bad setup by an administrator. Both NFS and CIFS can work with network-wide locks just fine. SQLite advises against using a networking file system to avoid potential issues, but you can successfully do it.

Are the typical Synology, Qnap, or TrueNAS devices with default Linux, macOS and Windows clients going to be set up correctly by default? If any of the typical things someone is likely to setup following wizards in a home or small office is likely to result in lock not working correctly for SQLite, then it is fair for them to warn against using it on a network file system. As an application format, you don't generall…

> As an application format, you don't generally expect people to be editing an ODF file at the same time though

Oh hell yes you do. Excel spreadsheets are notorious for people wanting to collaborate on them, and PowerPoint sheets come in close second. It used to be an absolute PITA but at least Office 365 makes the pains bearable.

Re: What If OpenDocument Used SQLite?

#66

Post by mschuster91 »

treyd wrote: post #49
ignoramous wrote: post #27

> SQLite database has a lot of capability, which this essay has only begun to touch upon. But hopefully this quick glimpse has convinced some readers that using an SQL database as an application file format is worth a second look. It really is. One of the experiments we have been doing currently to make bug reporting from Androids easier (and to an extent, reduce user frustration and fatigue) is to store app logs (un…

What kinds of queries are being done on the logs such that it makes sense to use sqlite instead of, like, just a ring buffer?

The problem they're alluding to, I think, isn't the query side, it's the creation side. adb logcat and logging in Android in general is one hell of a clusterfuck, not being helped by logging in Java being a PITA.

Re: What If OpenDocument Used SQLite?

#67

Post by bob1029 »

Retr0id wrote: post #29
liuliu wrote: post #24

One thing I would call out, if you use SQLite as an application format: BLOB type is limited to 2GiB in size (int32). Depending on your use cases, that might seem high, or not. People would argue that if you store that much of binary data in a SQLite database, it is not really appropriate. But, application format usually has this requirement to bundle large binary data in one nice file, rather than many files that yo…

You can split your data up across multiple blobs

This is essential if you want to have encryption/compression + range access at the same time.

I've been using chunk sizes of 128 megabytes for my media archive. This seems to be a reasonable tradeoff between range retrieval delay and per object overhead (e.g. s3 put/get cost).

Re: What If OpenDocument Used SQLite?

#68

Post by munch117 »

wzdd wrote: post #56
chris_wot wrote: post #38

Earlier quoted context omitted.

"Most applications can use SQLite without having to worry about bugs in obscure SQL inputs." And then they recommend SQLite as a document interchange format.

Although this is indeed a worrying statement, it seems true to me. Most users of sqlite control the SQL they use. The problem I would expect from using a database document interchange format is that a maliciously crafted database could result in a CVE. The page acknowledges this possibility, even while pointing out (in their CVE list) that it hasn't happened so far, or is rare (it's hard to parse some of their descri…

I'm not that concerned with bugs in sqlite. sqlite is high quality software, and the application that uses it is a more likely source of vulnerabilities.

But I do see a problem if you really need to use a sqlite that's compiled with particular non-default options.

Say I design a file format and implement it, and my implementation uses an sqlite library that's compiled with all the right options. Then I evangelize my file format, telling everyone that it's really just an sqlite database and sooo easy to work with.

First thing that happens is that someone writes a neat little utility for working with the files, written in language X, which comes with a handy sqlite3 library. But that library is not compiled with the right options, and boom, you have a vulnerable utility.

Re: What If OpenDocument Used SQLite?

#70

Post by thayne »

Ekaros wrote: post #62
jdboyd wrote: post #61

Earlier quoted context omitted.

Part of the problem though with saying SQLite instead of XML is a lot of things would lend themselves to XML in SQLite.

Complex features are inherently complex. Say you want external resources or some scripts in document. No matter what storage format you use those are more surfaces. Problem is not storage, but what is done with information. And very often that is a lot and poorly thought out and even more poorly implemented.

But most applications don't need those features. And if they do, that should be part of the application logic, with appropriate controls. Having your parsing library make arbitrary http requests is a bad idea.
Post reply on HN