Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite Is a Library of Congress Recommended Storage Format

sqlite.org

Post reply on HN
Print view View original

81–90 of 205 posts

Re: SQLite Is a Library of Congress Recommended Storage Format

#81

Post by keybored »

zoky wrote: post #38
tnelsond4 wrote: post #33

I'm always inspired by SQLite. Overall I like it, but if you're not doing writes it's really overkill. So I made a format that will never surpass SQLite, except that it's extremely lighter and faster and works on zstd compressed files. It has really small indexes and can contain binaries or text just like SQLite. The wasm part that decompresses and reads and searches the databases is only 38kb (uncompressed (maybe 16…

something something XKCD competing standards something something

Doesn’t even apply unless someone says that (1) there are too many “standards”, and (2) so we are making this standard (neither apply here). Someone made something.

We should really consider eventually retiring memes because they just end up as thought-terminating cliches.

This is of course referring to xkcd #927. How do I know that?

Re: SQLite Is a Library of Congress Recommended Storage Format

#82

Post by gpvos »

tnelsond4 wrote: post #33

I'm always inspired by SQLite. Overall I like it, but if you're not doing writes it's really overkill. So I made a format that will never surpass SQLite, except that it's extremely lighter and faster and works on zstd compressed files. It has really small indexes and can contain binaries or text just like SQLite. The wasm part that decompresses and reads and searches the databases is only 38kb (uncompressed (maybe 16…

A more standard solution would be cdb.[0] Although that doesn't support compressed data.

[0] https://cdb.cr.yp.to/ , https://en.wikipedia.org/wiki/Cdb_(software)

Re: SQLite Is a Library of Congress Recommended Storage Format

#84

Post by jjice »

llagerlof wrote: post #83

I used SQLite for a few applications several years ago. One time, the database got corrupted and all the data was lost. That was the day I stopped using SQLite. Also, the lack of enforced column data types was always a negative for me.

No matter the medium, backups are a must.

Re: SQLite Is a Library of Congress Recommended Storage Format

#85

Post by mr_toad »

Fwirt wrote: post #5
alexpotato wrote: post #2

I have always loved SQLite. I have also heard that some firms ban its use. Why? Because it makes it SO easy to set up a database for your app that you end up with a super critical component of your application that looks exactly like a file. A file that can have any extension. And that file can be copied around to other servers. Even if there is PII in that file. Multiply this times the number of applications in your…

The question is, do the same firms ban Excel? Excel spreadsheets often end up as shadow databases in unlikely places.

I’ve worked at some organisations that have strict rules (not always strictly followed) about what can go in Excel spreadsheets, and where they have to be stored. The C drive is verboten. Some also have standards about classification and labelling of PII and sensitive data.

Re: SQLite Is a Library of Congress Recommended Storage Format

#86

Post by sgbeal »

tnelsond4 wrote: post #33

I'm always inspired by SQLite. Overall I like it, but if you're not doing writes it's really overkill. So I made a format that will never surpass SQLite, except that it's extremely lighter and faster and works on zstd compressed files. It has really small indexes and can contain binaries or text just like SQLite. The wasm part that decompresses and reads and searches the databases is only 38kb (uncompressed (maybe 16…

> Compare that to SQLite's 1.2mb of wasm and glue code

The current trunk is actually 1.7mb in its canonical unminified form (which includes very nearly as much docs as JS code), split almost evenly between the WASM and JS pieces :/. Edit: it is 1.2mb in minified form, though.

Disclosure: i'm its maintainer.

Edit: current trunk, for the sake of trivia:

    sqlite3.wasm 896745
    sqlite3.mjs  816270 # unminified w/ docs
    sqlite3.mjs  431388 # unminified w/o docs
    sqlite3.mjs  310975 # minified

Re: SQLite Is a Library of Congress Recommended Storage Format

#88

Post by perching_aix »

DeathArrow wrote: post #21
Fwirt wrote: post #5

Earlier quoted context omitted.

The question is, do the same firms ban Excel? Excel spreadsheets often end up as shadow databases in unlikely places.

Do companies ban text files? Text files are used to store data.

That's why you store them on unsaved tabs instead.

Re: SQLite Is a Library of Congress Recommended Storage Format

#89

Post by perching_aix »

pletnes wrote: post #44
testermelon wrote: post #41

I'm surprised they included proprietary format that's de facto standard in profession or supported by multiple tools (.xls, .xlsx) in preferred section [1]. I wonder if "well-known enough" is as good as "open" from preservation standpoint. [1] https://www.loc.gov/preservation/resources/rfs/data.html

You can unzip the xlsx and read the xml inside. It’s not the worst format by far.

What would you reckon is the worst format? I'm very curious of your standards given this.

Re: SQLite Is a Library of Congress Recommended Storage Format

#90

Post by smartmic »

tnelsond4 wrote: post #33

I'm always inspired by SQLite. Overall I like it, but if you're not doing writes it's really overkill. So I made a format that will never surpass SQLite, except that it's extremely lighter and faster and works on zstd compressed files. It has really small indexes and can contain binaries or text just like SQLite. The wasm part that decompresses and reads and searches the databases is only 38kb (uncompressed (maybe 16…

Many comments here to your creation, PeakSlab, but not yet a dedicated praise. I didn't know it but I have to say it is really cool and innovative! The performance of the dictionary is indeed superb and I will definitely bookmark this for future reuse. So, in a nutshell: thanks for sharing!
Post reply on HN