Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What if OpenDocument used SQLite? (2014)

sqlite.org

Post reply on HN
Print view View original

291–300 of 310 posts

Re: What if OpenDocument used SQLite? (2014)

#291

Post by gwbas1c »

Tempest1981 wrote: post #214
gwbas1c wrote: post #135

I shipped a product that used both SQLite and XML files. One of the improvements that I made was moving a few tables that contained small amounts of data to xml files. Because these files were small and rarely written; it simplified the data access layer, and simplified diagnostics. (I made sure the files were multi-line tabbed xml.) For "technical" people who needed to diagnose the product, asking them to crack open…

I'm unclear on how SQLite (native format, no zip) is achieving sizes similar to XML + Zip. Are SQLite TEXT or BLOB fields compressed? Or are they assuming the caller is compressing BLOBs before writing?

SQLite does not compress, as far as I know.

Engineering is all about tradeoffs: SQLite is optimized for quick incremental updates where you don't need to rewrite the whole file. Zip & xml aren't. (IE, if you decide to add a letter to a word at the beginning of a document, with zip & XML you have to rewrite the whole document. SQLite can make a minor change without the whole rewrite.)

In our case, file size was not a factor in choosing between SQLite and XML.

But, remember that file size is deceptive: Disks are block devices; the 30 byte and 1k file take up the same space if you block size is 2k. (I've shipped a filesystem driver.) HTTP servers gzip on download. It's more important to know your needs than to get hung up on a single metric like file size.

> I'm unclear on how SQLite (native format, no zip) is achieving sizes similar to XML + Zip. Are SQLite TEXT or BLOB fields compressed? Or are they assuming the caller is compressing BLOBs before writing?

Remember, XML writes each tag name 1 time if there's no content and twice if there is. Each attribute has it's name written every time. I doubt SQLite writes all the metadata in each row.

Re: What if OpenDocument used SQLite? (2014)

#292

Post by mixmastamyk »

skybrian wrote: post #282
mixmastamyk wrote: post #222

Earlier quoted context omitted.

It's pretty rare to put office docs into version control, as they are typically binary instead of text. So, doesn't work well. Perhaps there is a version of open-doc that doesn't use the zip file but a folder of XML instead? Also the XML might need to be optimized to prefer line-oriented operations.

Yes, it's rare to use git, but it's also pretty well-known that people can share more than they intended in a Word document. Perhaps true of Open Office as well? See: https://superuser.com/questions/1562130/can-people-see-the-c... https://foiassist.ca/2019/04/04/i-thought-we-deleted-that-me...

Don't forget Exif, its thumbnail and even reflections, in photos.

Re: What if OpenDocument used SQLite? (2014)

#293

Post by sdeframond »

bane wrote: post #285
sdeframond wrote: post #246

Earlier quoted context omitted.

I think this has been discussed before about WebSQL. > The [WebSQL] specification reached an impasse: all interested implementors have used the same SQL backend (Sqlite), but we need multiple independent implementations to proceed along a standardisation path. https://www.w3.org/TR/webdatabase/

In other words, "all the implementors chose a standard, but we're the standard deciders so we're killing the whole idea".

The standard deciders are the implementors. There is no point in opposing them. The W3C is actually the representatives of Google, Mozilla, Microsoft, Opera and so on.

Re: What if OpenDocument used SQLite? (2014)

#294

Post by rini17 »

mcpackieh wrote: post #239
rini17 wrote: post #11

Earlier quoted context omitted.

Yes it should replicate the functionality user expects - save everything into temporary file and overwrite the original file only on explicit save action. As for Git, it would benefit from using text format specifically aimed for easy diffing/merging. No idea how easy the sqlite dump is in this regard.

I have been told that the new generation of users does not expect, want or appreciate applications that use explicit saves. I've also been told that they don't understand or even want to understand folders...

The context here is someone using git. Who presumably understands folders.

Re: What if OpenDocument used SQLite? (2014)

#295

Post by eternityforest »

miki123211 wrote: post #136

The problem with SQLite is that it's not a standardized file format. It's well-documented and pretty well understood for sure, but there's no ISO standard defining how to interpret an SQLite file in excruciating detail. Same goes for competing implementations, Zip and XML have a much smaller API surface than SQLite, whose API, apart from a bunch of C functions, is the SQL language itself. Writing an XML parser is not…

It almost seems worth giving up ISO for SQLite, but I understand there are real concerns when you get into enterprisy stuff.

SQLite is kind of its own standard. It's public domain and they don't do breaking changes all day, and it's in C. As long as C is still viable, SQLite is usable on basically all non embedded platforms, and nobody really needs to reimplement it, unless they want to port it to Rust or something.

Not that you'd need to, since it's already very reliable.

Re: What if OpenDocument used SQLite? (2014)

#296

Post by eternityforest »

fluidcruft wrote: post #116
rakoo wrote: post #86

Earlier quoted context omitted.

Why do you use a secondary, volatile database ? Performance-wise you won't gain a lot more (we're talking about a user editing a file, so not even 1 write per second). A proposal: write directly, and automatically in the database. No more Save button. There are multiple advantages: - the system is crash-resistant. I like taking the approach of CouchDB where the only correct way to close the system is to crash it. Tha…

There is nothing I hate more than an app that modifies files secretly when I open them. Then I have to get all defensive to copy files before I open them to keep them intact. You may not see the problem with changing the checksum or hash of a file, but silently tampering with files is a nightmare in many domains. If you open a file and accidentally change something trivial (some apps like to store things like present…

There's also SSD wear issues. There shouldn't be, because SSDs are durable, but some applications find dumb reasons to write multiple GB in a minute.

And by some applications I pretty much just mean browsers, but still.

Re: What if OpenDocument used SQLite? (2014)

#297

Post by orf »

hot_gril wrote: post #289
orf wrote: post #284

Earlier quoted context omitted.

I’d love to understand your thinking behind the idea that a document standard should rely on a query language and not a file format… Document standards are file formats… Or are you saying a document format should just be some DDL statements? What? How is that interoperable? It’s coupled to the database that is storing the data as an implementation detail , which is exactly the problem with using SQLite . > That alone…

> Or are you saying a document format should just be some DDL statements? What? How is that interoperable? Yes. How is it interoperable, because it's quite easy to make DDL for SQLite that also works for many other DBMSes, given that SQLite is kinda the lowest common denominator of those. Maybe not as interoperable as ODF since it's easier to implement an ODF parser/writer than a SQLite clone, but probably more inter…

> Maybe not as interoperable as ODF since it's easier to implement an ODF parser/writer than a SQLite clone

Ladies and gentlemen: he’s so close, he’s nearly there, but he just can’t make the final connection!

Re: What if OpenDocument used SQLite? (2014)

#298

Post by euroderf »

iefbr14 wrote: post #44
pgeorgi wrote: post #41

Earlier quoted context omitted.

WinFS ( https://en.wikipedia.org/wiki/WinFS ) without the mssql Engine?

Or this: https://github.com/narumatt/sqlitefs

And of course there is sqlarfs, at the bottom of https://www.sqlite.org/sqlar/doc/trunk/README.md

Re: What if OpenDocument used SQLite? (2014)

#299

Post by hot_gril »

p4bl0 wrote: post #6

I'm currently working on an application where I use SQLite as the file format. I want to keep a usual workflow for users where you can make edit to your document and it only changes the file when you save it. So to open a file I copy it into the :memory: database [1], then the user can do whatever manipulation they want and I can directly make the change in the database I don't need to have a model of the document ot…

Btw, Apple's CoreData, commonly used by iPhone and Mac apps, uses SQLite by default. That part works fine, so you can study it if you'd like and ignore all the bad parts built on top (ORM, MVC framework, etc).

Re: What if OpenDocument used SQLite? (2014)

#300

Post by Eduard »

cornstalks wrote: post #252
Eduard wrote: post #244

Earlier quoted context omitted.

funny, the RFC even includes a shell command pipeline to extract the base64 out of the awkward RFC formatting. Using the C source code still leaves room for ambiguities / under-specification, no? After all, the semantics rely on the particular gcc release used for compiling the code.

There is still the possibility of a bug or under-specification, but that's always the case in any spec. At least with Opus they document what implementation-defined behavior they require, so assuming there aren't any hidden bugs then you should get consistent output across compilers.

but the semantics change depending on the build tool version and other factors.
Post reply on HN