Live data from Hacker News

SQLite Is a Library of Congress Recommended Storage Format

sqlite.org

41–50 of 205 posts

Re: SQLite Is a Library of Congress Recommended Storage Format

#41
I'm surprised they included proprietary format that's de facto standard in profession or supported by multiple tools (.xls, .xlsx) in preferred section [1]. I wonder if "well-known enough" is as good as "open" from preservation standpoint.

[1] https://www.loc.gov/preservation/resources/rfs/data.html

Re: SQLite Is a Library of Congress Recommended Storage Format

#42
post #27
post #22

On a recent project I have needed to use exFAT. exFAT is terrible for a number of reasons, but in my case the thing I had to deal with was the lack of journaling, which had the possibility to corrupt files if there were a power interruption or something. I initially was writing a series of files and doing some quasi-append-only things with new files and compacting the old one to sort of reinvent journaling. What I di…

The problem with it is you didn't solve your biggest actual problem, you just haven't had a problem bite you in the ass yet so you think your problem is solved.

I am not sure the problem is actually fully solvable. I think SQLite helps at least a little.

Re: SQLite Is a Library of Congress Recommended Storage Format

#43
post #26
post #22

On a recent project I have needed to use exFAT. exFAT is terrible for a number of reasons, but in my case the thing I had to deal with was the lack of journaling, which had the possibility to corrupt files if there were a power interruption or something. I initially was writing a series of files and doing some quasi-append-only things with new files and compacting the old one to sort of reinvent journaling. What I di…

> I wish exFAT would die in a fire and a journaling filesystem would replace it as the "one filesystem you can use everywhere" Where exactly is everywhere? Win32? All of Linux? BSDs? MacOS? IOS? ...

Everywhere exFAT is supported now. Windows, Mac, Linux, FreeBSD would be fine.

Re: SQLite Is a Library of Congress Recommended Storage Format

#44

I'm surprised they included proprietary format that's de facto standard in profession or supported by multiple tools (.xls, .xlsx) in preferred section [1]. I wonder if "well-known enough" is as good as "open" from preservation standpoint. [1] https://www.loc.gov/preservation/resources/rfs/data.html

You can unzip the xlsx and read the xml inside. It’s not the worst format by far.

Re: SQLite Is a Library of Congress Recommended Storage Format

#45

I'm surprised they included proprietary format that's de facto standard in profession or supported by multiple tools (.xls, .xlsx) in preferred section [1]. I wonder if "well-known enough" is as good as "open" from preservation standpoint. [1] https://www.loc.gov/preservation/resources/rfs/data.html

Especially when Office 365 shows that not even Microsoft is capable of making software which can display Office files anymore... if you have a Word file which was created or has ever been modified by the Word application, working with it through Office 365 in a browser is such a pain. I've literally had images which are impossible to delete or move in the web version, and they will absolutely render in the wrong place.

Re: SQLite Is a Library of Congress Recommended Storage Format

#47

I'm always inspired by SQLite. Overall I like it, but if you're not doing writes it's really overkill. So I made a format that will never surpass SQLite, except that it's extremely lighter and faster and works on zstd compressed files. It has really small indexes and can contain binaries or text just like SQLite. The wasm part that decompresses and reads and searches the databases is only 38kb (uncompressed (maybe 16…

Perhaps a dumb question, but how do you get data into it if you’re not doing writes

Re: SQLite Is a Library of Congress Recommended Storage Format

#48
post #5

Earlier quoted context omitted.

The question is, do the same firms ban Excel? Excel spreadsheets often end up as shadow databases in unlikely places.

The sane thing would be to ban Excel and promote SQLite. Excel is often used for tabulated text (issue tracking) not calculations. Perfect use case for a relational db

Excel has sheets for tables, columns and rows, primary keys (UNIQUE), foreign key references etc if you squint.

It doesn't require you use all of that properly, but it's there.

Re: SQLite Is a Library of Congress Recommended Storage Format

#49
post #47

I'm always inspired by SQLite. Overall I like it, but if you're not doing writes it's really overkill. So I made a format that will never surpass SQLite, except that it's extremely lighter and faster and works on zstd compressed files. It has really small indexes and can contain binaries or text just like SQLite. The wasm part that decompresses and reads and searches the databases is only 38kb (uncompressed (maybe 16…

Perhaps a dumb question, but how do you get data into it if you’re not doing writes

Generate it one time from a source tsv file or folder of media.

Re: SQLite Is a Library of Congress Recommended Storage Format

#50
post #5

Earlier quoted context omitted.

The question is, do the same firms ban Excel? Excel spreadsheets often end up as shadow databases in unlikely places.

The sane thing would be to ban Excel and promote SQLite. Excel is often used for tabulated text (issue tracking) not calculations. Perfect use case for a relational db

and excel has gui for forms
Post reply on HN