Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite Is a Library of Congress Recommended Storage Format

sqlite.org

Post reply on HN
Print view View original

61–70 of 205 posts

Re: SQLite Is a Library of Congress Recommended Storage Format

#61

Post by [deleted] »

hparadiz wrote: post #59
0123456789ABCDE wrote: post #57

Earlier quoted context omitted.

all* of that + sharding -> https://sqlite.org/lang_attach.html ex: main.db + fts.db. reading and writing to main.db is always available; updating the fts index can be done without blocking the main database — it only needs to read, the reads can be chunked, and delayed. fts.db keeps the index + a cursor table — an id or last change ts could also use a shard to handle tables for metrics, or simply move old data out of…

Oh fun something I have some metrics on. I just made this benchmark for every php orm a few weeks ago for fun. https://the-php-bench.technex.us/ There's a huge performance difference between memory and file storage within sqlite itself. Not even getting into tuning specifics.

[deleted]

Re: SQLite Is a Library of Congress Recommended Storage Format

#62

Post by rantingdemon »

0123456789ABCDE wrote: post #50
hermitShell wrote: post #6

Earlier quoted context omitted.

The sane thing would be to ban Excel and promote SQLite. Excel is often used for tabulated text (issue tracking) not calculations. Perfect use case for a relational db

and excel has gui for forms

Only where VBA is available. Not available for MacOs versions if I'm correct?

Re: SQLite Is a Library of Congress Recommended Storage Format

#63

Post by pbhjpbhj »

tombert wrote: post #43
mmooss wrote: post #26

Earlier quoted context omitted.

> I wish exFAT would die in a fire and a journaling filesystem would replace it as the "one filesystem you can use everywhere" Where exactly is everywhere? Win32? All of Linux? BSDs? MacOS? IOS? ...

Everywhere exFAT is supported now. Windows, Mac, Linux, FreeBSD would be fine.

Presumably Microsoft fear making it easy to swap OSes and access the same data.

"I can use Linux because if I get stuck I can just switch to Windows and still access my data" is a comfort that probably keeps people from even trying Linux (or other OSes)?

Why else would MS not support BTRFS/ZFS/Ext or whatever?

{I'm not saying that I think this works.}

Re: SQLite Is a Library of Congress Recommended Storage Format

#64

Post by tnelsond4 »

xxs wrote: post #39
maxloh wrote: post #32

Earlier quoted context omitted.

FYI, they added a lot more formats to the list after that. Preferred 1. Platform-independent, character-based formats are preferred over native or binary formats as long as data is complete, and retains full detail and precision. Preferred formats include well-developed, widely adopted, de facto marketplace standards, e.g. a. Formats using well known schemas with public validation tool available b. Line-oriented, e.g…

.7z being there just discredits the entire process. The underlying compression algorithm is a free-hand one and can be anything[0], or contain bugs and exploits[1]. Personally I use only zstd with .7z which is 'non-standard' by the official (Russian) release. [0]: https://7-zip.org/7z.html [1]: CVE-2025-0411

I love using zstd, it's so fast to decompress. I especially like that the JavaScript decoder is 8kb and still really fast. Though the 25kb wasm decoders are about twice as fast.

What are the advantages or reasons to use zstd in a 7z container versus just .zst?

Re: SQLite Is a Library of Congress Recommended Storage Format

#65

Post by harvie »

hermitShell wrote: post #6
Fwirt wrote: post #5

Earlier quoted context omitted.

The question is, do the same firms ban Excel? Excel spreadsheets often end up as shadow databases in unlikely places.

The sane thing would be to ban Excel and promote SQLite. Excel is often used for tabulated text (issue tracking) not calculations. Perfect use case for a relational db

or reimplement excel with sqlite as a backend :-D

BTW sqlite can run SQL queries on CSV files with relatively simple one-liner command...

Re: SQLite Is a Library of Congress Recommended Storage Format

#66

Post by iknowstuff »

pbhjpbhj wrote: post #63
tombert wrote: post #43

Earlier quoted context omitted.

Everywhere exFAT is supported now. Windows, Mac, Linux, FreeBSD would be fine.

Presumably Microsoft fear making it easy to swap OSes and access the same data. "I can use Linux because if I get stuck I can just switch to Windows and still access my data" is a comfort that probably keeps people from even trying Linux (or other OSes)? Why else would MS not support BTRFS/ZFS/Ext or whatever? {I'm not saying that I think this works.}

> Why else would MS not support BTRFS/ZFS/Ext or whatever?

You seriously can’t think of another reason? File systems are complex. Maintenance is a huge burden. Getting them wrong is a liability. Reason enough to only support the bare minimum. And then, 99% of their users don’t care about any of those. NTFS is good enough

Re: SQLite Is a Library of Congress Recommended Storage Format

#67

Post by andrelaszlo »

giza182 wrote: post #47
tnelsond4 wrote: post #33

I'm always inspired by SQLite. Overall I like it, but if you're not doing writes it's really overkill. So I made a format that will never surpass SQLite, except that it's extremely lighter and faster and works on zstd compressed files. It has really small indexes and can contain binaries or text just like SQLite. The wasm part that decompresses and reads and searches the databases is only 38kb (uncompressed (maybe 16…

Perhaps a dumb question, but how do you get data into it if you’re not doing writes

I think it's just immutable once you've generated it. No need to update indexes or check consistency on writes, no need for transactions, etc.

Re: SQLite Is a Library of Congress Recommended Storage Format

#68

Post by harrouet »

tehlike wrote: post #17
rmunn wrote: post #16

> As of this writing (2018-05-29) ... So this news is nearly six EIGHT years old. But I didn't happen to know about it until now, so that's not a complaint at all; rather, this is a thank-you for posting it. (Thanks for the correction. Brief brain malfunction in the math department there).

Sir, it's 2026. It's 8 years old.

Not if the GP was written 2 years ago :)
Post reply on HN