Live data from Hacker News

SQLite Is a Library of Congress Recommended Storage Format

sqlite.org

91–100 of 205 posts

Re: SQLite Is a Library of Congress Recommended Storage Format

#91
post #31

Earlier quoted context omitted.

IMO, almost any Excel more than a month old should become readonly.

You should consider knock-on effects of this brilliant idea. Now there would be copies of spreadsheets younger than a month that get replicated 47 billion times, exponentially compounding the problem you're trying to solve. This sounds like how we pass so many stupid laws. Nobody thinks about 2nd order effects.

So you're saying they should further auto-delete after two or three months?

Re: SQLite Is a Library of Congress Recommended Storage Format

#92
post #39

Earlier quoted context omitted.

.7z being there just discredits the entire process. The underlying compression algorithm is a free-hand one and can be anything[0], or contain bugs and exploits[1]. Personally I use only zstd with .7z which is 'non-standard' by the official (Russian) release. [0]: https://7-zip.org/7z.html [1]: CVE-2025-0411

I love using zstd, it's so fast to decompress. I especially like that the JavaScript decoder is 8kb and still really fast. Though the 25kb wasm decoders are about twice as fast. What are the advantages or reasons to use zstd in a 7z container versus just .zst?

I love zstd as the next guy and I do use zstd solo for the most part. I had a talk on it few years back too (incl. using the lib directly from Java, massively decreasing log storage, and so on).

Why use it w/ 7-zip though. 7-zip archives multiple files/directories and supports encryption. It has the UI too.. On Windows there is NanaZip that's available in the microsoft store which has been graced by corporate for user-install (unlike zstd that effectively needs WSL), and most folks won't be able to use the command line tool.

Of course using tar with zstd is always an option if you are on linux.

Re: SQLite Is a Library of Congress Recommended Storage Format

#93
post #4

2026 recommended storage formats: https://www.loc.gov/preservation/resources/rfs/data.html

Taking a minute to appreciate the level of long term thinking required for storing data, to plan for 300-500 years into the future, to be able to withstand all kinds of innovations, and survive basic obsolescence.

What is the longest surviving paper medium?

Re: SQLite Is a Library of Congress Recommended Storage Format

#94

Earlier quoted context omitted.

You should consider knock-on effects of this brilliant idea. Now there would be copies of spreadsheets younger than a month that get replicated 47 billion times, exponentially compounding the problem you're trying to solve. This sounds like how we pass so many stupid laws. Nobody thinks about 2nd order effects.

So you're saying they should further auto-delete after two or three months?

3rd order effect, people copy and paste the old sheet into a new sheet, now we have worse exponential. You’re not very good at this huh.

Re: SQLite Is a Library of Congress Recommended Storage Format

#95

I'm surprised they included proprietary format that's de facto standard in profession or supported by multiple tools (.xls, .xlsx) in preferred section [1]. I wonder if "well-known enough" is as good as "open" from preservation standpoint. [1] https://www.loc.gov/preservation/resources/rfs/data.html

Archivists and librarians have to think in terms of practicality: if many tools exist to read something and it’s a mainstream software product, the odds are good that they’ll be unable to use those files 50 years from now. Not certain, but good, and that matters with limited budget and ability to tell the rest of the world what format to provide things in.

This can require nuance: for example, PDF has profiles because the core format is widely supported but you could do things like embed plugin content from now-defunct vendors and they would only want the former for long-term preservation.

Re: SQLite Is a Library of Congress Recommended Storage Format

#96

Earlier quoted context omitted.

So you're saying they should further auto-delete after two or three months?

3rd order effect, people copy and paste the old sheet into a new sheet, now we have worse exponential. You’re not very good at this huh.

Which is very annoying and people will complain. People complaining can be then directed towards a better solution. As a bonus, mistakes will also rise, leading to further complaints, especially ones that reach higher. All this making the dogshit practice, and the idiots committing them, infinitely more visible and thus fixable.

The sheer volume of data that needs tending to may even grind certain departments to a halt! What a great opportunity! It'd appear I'm positively stellar at this!

Re: SQLite Is a Library of Congress Recommended Storage Format

#97
post #55

I get annoyed at all the other DBs that require their own heavy duty server process when for 90% of my projects there is only one client, my app server. Is there a DB that combines sqlite's embedded simplicity with higher concurrent write throughput?

I think the concurrent write thing is not as much of an issue nowadays with the speed of NVMEs and WAL.

Re: SQLite Is a Library of Congress Recommended Storage Format

#98

I used SQLite for a few applications several years ago. One time, the database got corrupted and all the data was lost. That was the day I stopped using SQLite. Also, the lack of enforced column data types was always a negative for me.

> the database got corrupted

What caused that?

Re: SQLite Is a Library of Congress Recommended Storage Format

#99

I used SQLite for a few applications several years ago. One time, the database got corrupted and all the data was lost. That was the day I stopped using SQLite. Also, the lack of enforced column data types was always a negative for me.

For column types there are STRICT tables now

Re: SQLite Is a Library of Congress Recommended Storage Format

#100

I'm always inspired by SQLite. Overall I like it, but if you're not doing writes it's really overkill. So I made a format that will never surpass SQLite, except that it's extremely lighter and faster and works on zstd compressed files. It has really small indexes and can contain binaries or text just like SQLite. The wasm part that decompresses and reads and searches the databases is only 38kb (uncompressed (maybe 16…

SQLite is simple in its own way and I like the design principle of their SQL dialect.

"Right joins are just left joins in the wrong direction, you don't need that crap"

Of course it always gets simpler or more specialised. I think many apps using databases would run with SQLite just as well. And some would probably run just as well with a textfile instead of any db like SQLite.

Post reply on HN