Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What if OpenDocument used SQLite? (2014)

sqlite.org

Post reply on HN
Print view View original

281–290 of 310 posts

Re: What if OpenDocument used SQLite? (2014)

#281

Post by pl4nty »

btilly wrote: post #231
bane wrote: post #157

Earlier quoted context omitted.

I'm not sure if the problem you are pointing out has to do with: a) SQLite the file format - which is Public Domain and so well documented that parsers for it exist in numerous other languages even though it's almost pointless because... b) SQLite, the Public Domain (and thus entirely source available) C implementation of the library that can operate on the file format -- and is documented to a level well above what…

It is possible that the C implementation of SQLite is the single most commonly deployed software library ever. If not, then it is probably the second, after zlib. https://www.sqlite.org/mostdeployed.html Therefore I consider it a better supported format than most standardized formats.

for one, it's bundled with consumer versions of Windows as winsqlite3.dll. not sure when this started though

Re: What if OpenDocument used SQLite? (2014)

#282

Post by skybrian »

mixmastamyk wrote: post #222
skybrian wrote: post #219

Implementing versioning in the file format conflicts with git, because each document is essentially its own little source control system. This can be surprising to users who copy the file and don’t realize that they’ve effectively copied the entire repo. Copying a file will sometimes include drafts they didn’t want to share. It can mean you lose control over when things are committed, and so you don’t end up with a u…

It's pretty rare to put office docs into version control, as they are typically binary instead of text. So, doesn't work well. Perhaps there is a version of open-doc that doesn't use the zip file but a folder of XML instead? Also the XML might need to be optimized to prefer line-oriented operations.

Yes, it's rare to use git, but it's also pretty well-known that people can share more than they intended in a Word document. Perhaps true of Open Office as well? See:

https://superuser.com/questions/1562130/can-people-see-the-c...

https://foiassist.ca/2019/04/04/i-thought-we-deleted-that-me...

Re: What if OpenDocument used SQLite? (2014)

#283

Post by hot_gril »

orf wrote: post #280
hot_gril wrote: post #279

Earlier quoted context omitted.

I don't know, probably a lot of us have dealt with large docs that become noticeably slow to edit and scary to save, mostly spreadsheets.

Ok cool: so adding SQL to that is going to magically speed it up? No. It’s the on disk format that matters. Because it would be just as slow and scary if it used a sqlite file that was embedded in a zip file or something equally as mad. It’s not the SQL, it’s the file format. If you decouple the file format from the SQL engine, it becomes simpler to reimplement, more agnostic and less vendor locked.

SQLite (not just the language SQL) would make it much easier to reimplement in a way that's fast and safe, yes.

> If you decouple the file format from the SQL engine

That alone would be a difficult project. If you really want to break things down, easier to say your document standard relies on SQLite's rather simple query language (https://www.sqlite.org/lang.html), and there, it's independent of SQLite's query planner and file format. Wouldn't be hard to make it work with Postgres or MySQL, for instance.

Re: What if OpenDocument used SQLite? (2014)

#284

Post by orf »

hot_gril wrote: post #283
orf wrote: post #280

Earlier quoted context omitted.

Ok cool: so adding SQL to that is going to magically speed it up? No. It’s the on disk format that matters. Because it would be just as slow and scary if it used a sqlite file that was embedded in a zip file or something equally as mad. It’s not the SQL, it’s the file format. If you decouple the file format from the SQL engine, it becomes simpler to reimplement, more agnostic and less vendor locked.

SQLite (not just the language SQL) would make it much easier to reimplement in a way that's fast and safe, yes. > If you decouple the file format from the SQL engine That alone would be a difficult project. If you really want to break things down, easier to say your document standard relies on SQLite's rather simple query language ( https://www.sqlite.org/lang.html ), and there, it's independent of SQLite's query pla…

I’d love to understand your thinking behind the idea that a document standard should rely on a query language and not a file format…

Document standards are file formats…

Or are you saying a document format should just be some DDL statements? What? How is that interoperable? It’s coupled to the database that is storing the data as an implementation detail, which is exactly the problem with using SQLite.

> That alone would be a difficult project

I’m not suggesting using the SQLite file format, I’m suggesting the pretty basic idea that the storage for a general purpose widely used and interoperable document format should be logically decoupled from anything else, and definitely not be tied to the implantation details of a single library or even a single version of that library.

The file format is the most important part. It’s the only part. Nothing else matters because there is nothing else.

It’s not rocket science.

Re: What if OpenDocument used SQLite? (2014)

#285

Post by bane »

sdeframond wrote: post #246
bane wrote: post #157

Earlier quoted context omitted.

I'm not sure if the problem you are pointing out has to do with: a) SQLite the file format - which is Public Domain and so well documented that parsers for it exist in numerous other languages even though it's almost pointless because... b) SQLite, the Public Domain (and thus entirely source available) C implementation of the library that can operate on the file format -- and is documented to a level well above what…

I think this has been discussed before about WebSQL. > The [WebSQL] specification reached an impasse: all interested implementors have used the same SQL backend (Sqlite), but we need multiple independent implementations to proceed along a standardisation path. https://www.w3.org/TR/webdatabase/

In other words, "all the implementors chose a standard, but we're the standard deciders so we're killing the whole idea".

Re: What if OpenDocument used SQLite? (2014)

#286

Post by dragonwriter »

bane wrote: post #285
sdeframond wrote: post #246

Earlier quoted context omitted.

I think this has been discussed before about WebSQL. > The [WebSQL] specification reached an impasse: all interested implementors have used the same SQL backend (Sqlite), but we need multiple independent implementations to proceed along a standardisation path. https://www.w3.org/TR/webdatabase/

In other words, "all the implementors chose a standard, but we're the standard deciders so we're killing the whole idea".

One problem was the standard was bug-for-bug replication of a particular version of SQLite.

There’s very good reason for that not to be a standard. (Now, assuming the SQLite documentation is licensed in a way which supports this, copying the documentation of SQLite’s supported SQL as of that version into the standard might have been viable, but no one interested in having WebSQL proposed that or any other resolution.

That relates to the cited issue of absence of independent implementations, which would have been a problem even with a spec that supported independent implementations and verification of their compliance independent of a particular reference implementation. though I personally think the spec problem is a bigger real problem (even if not the decisive policy problem) than the “everyone is using the same underlying software to implement the spec” problem is in this case, where the shared implementation is a permissively licensed open source implementation sponsored by several of the browser vendors, among others.

Re: What if OpenDocument used SQLite? (2014)

#287

Post by bane »

dragonwriter wrote: post #286
bane wrote: post #285

Earlier quoted context omitted.

In other words, "all the implementors chose a standard, but we're the standard deciders so we're killing the whole idea".

One problem was the standard was bug-for-bug replication of a particular version of SQLite. There’s very good reason for that not to be a standard. (Now, assuming the SQLite documentation is licensed in a way which supports this, copying the documentation of SQLite’s supported SQL as of that version into the standard might have been viable, but no one interested in having WebSQL proposed that or any other resolution.…

hmmm...I appreciate the thoughtful reply. You bring up an interesting point. What is the SQLite documentation licensed as? I would assume PD like the rest of it, but I don't know that for sure.

Re: What if OpenDocument used SQLite? (2014)

#288

Post by justinclift »

regularfry wrote: post #253
justinclift wrote: post #172

Earlier quoted context omitted.

> ... that should make Audacity crash when it tries to corrupt the data. That'd be fairly non-optimal behaviour. ;) When the application tries to add wrong data (eg duplicate key violating uniqueness constraint), SQLite will return an error. The application should handle things better than by crashing. In theory anyway. :)

If your choices are a crash or corruption, choose the crash.

Sure, data integrity is important.

But hopefully the choices are a bit better than just those two. :)

Re: What if OpenDocument used SQLite? (2014)

#289

Post by hot_gril »

orf wrote: post #284
hot_gril wrote: post #283

Earlier quoted context omitted.

SQLite (not just the language SQL) would make it much easier to reimplement in a way that's fast and safe, yes. > If you decouple the file format from the SQL engine That alone would be a difficult project. If you really want to break things down, easier to say your document standard relies on SQLite's rather simple query language ( https://www.sqlite.org/lang.html ), and there, it's independent of SQLite's query pla…

I’d love to understand your thinking behind the idea that a document standard should rely on a query language and not a file format… Document standards are file formats… Or are you saying a document format should just be some DDL statements? What? How is that interoperable? It’s coupled to the database that is storing the data as an implementation detail , which is exactly the problem with using SQLite . > That alone…

> Or are you saying a document format should just be some DDL statements? What? How is that interoperable?

Yes. How is it interoperable, because it's quite easy to make DDL for SQLite that also works for many other DBMSes, given that SQLite is kinda the lowest common denominator of those.

Maybe not as interoperable as ODF since it's easier to implement an ODF parser/writer than a SQLite clone, but probably more interoperable than some kind of advanced ODF designed for efficient updates. Just because you define a standard doesn't mean there are good portable implementations out there.

Re: What if OpenDocument used SQLite? (2014)

#290

Post by throwaway894345 »

SQLite wrote: post #208
throwaway894345 wrote: post #99

Is SQLite’s disk format an open, versioned standard? Or is it just “however SQLite saves data to disk”?

SQLite file format spec: https://www.sqlite.org/fileformat2.html Complete version history: https://sqlite.org/docsrc/finfo/pages/fileformat2.in Note that there have been no breaking changes since the file format was designed in 2004. The changes shows in the version history above have all be one of (1) typo fixes, (2) clarifications, or (3) filling in the "reserved for future extensions" bits with descriptions of tho…

Thanks for elaborating so thoroughly. I didn’t even realize you were on this platform!
Post reply on HN