Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite Is a Library of Congress Recommended Storage Format

sqlite.org

Post reply on HN
Print view View original

191–200 of 205 posts

Re: SQLite Is a Library of Congress Recommended Storage Format

#191

Post by tnelsond4 »

tracker1 wrote: post #190
tnelsond4 wrote: post #33

I'm always inspired by SQLite. Overall I like it, but if you're not doing writes it's really overkill. So I made a format that will never surpass SQLite, except that it's extremely lighter and faster and works on zstd compressed files. It has really small indexes and can contain binaries or text just like SQLite. The wasm part that decompresses and reads and searches the databases is only 38kb (uncompressed (maybe 16…

I've had to deal with image scans and my own approach lately has been to quantize to 16-color (grayscale via pngquant) and oxipng with zopfli set to max. Mostly so that the output can be easily used in a browser UI. Will definitely have to look into your solution with JBIG2 next time. Though I may keep with png just for compatibility sake.

Yeah, I had Claude generate me the jbig2 to 1-bit png wasm module from the jbig2 source so that there'd be no dependencies on zlib or libpng. Then I aggressively removed functions from 97kb down to 17kb. So if it's missing functions you need you may need to reset the files to the .bak and recompile.

Re: SQLite Is a Library of Congress Recommended Storage Format

#192

Post by darthwalsh »

alexpotato wrote: post #2

I have always loved SQLite. I have also heard that some firms ban its use. Why? Because it makes it SO easy to set up a database for your app that you end up with a super critical component of your application that looks exactly like a file. A file that can have any extension. And that file can be copied around to other servers. Even if there is PII in that file. Multiply this times the number of applications in your…

A production app with customer data needs a data backup/restore strategy. I'm guessing a random app server writing to a local sqlite file isn't doing that either.

Re: SQLite Is a Library of Congress Recommended Storage Format

#193

Post by xenadu02 »

ashellunts wrote: post #146
faangguyindia wrote: post #23

I went from thinking “SQLite is a toy product, not reliable for real data" to "lets use SQLite for almost everything" SQLite is very good if you can fit into the single writer, multiple readers pattern; you'll never lose data if you use the correct settings, which takes a minute of Google search to figure out. Today, most of my apps are simply go binary + SQLite + systemd service file. I've yet to lose data. Performa…

Do you use multiple backend nodes? If yes, how do you access sqlite files from different nodes?

You can always route writes to a writer node with streaming WAL replication to all the reader nodes. Works for some workloads and systems, not for others.

For that matter if you write your system with the correct abstraction you can switch to Postgres _later_ if it becomes necessary. For every system that really did need to scale 10,000 are pointlessly overbuilt - worrying about scale when it just didn't matter.

Re: SQLite Is a Library of Congress Recommended Storage Format

#196

Post by silon42 »

irishcoffee wrote: post #80
silon42 wrote: post #31

Earlier quoted context omitted.

IMO, almost any Excel more than a month old should become readonly.

You should consider knock-on effects of this brilliant idea. Now there would be copies of spreadsheets younger than a month that get replicated 47 billion times, exponentially compounding the problem you're trying to solve. This sounds like how we pass so many stupid laws. Nobody thinks about 2nd order effects.

Doesn't this happen anyway with "final2.really.final" ?

Re: SQLite Is a Library of Congress Recommended Storage Format

#198

Post by redeeman »

Scribbd wrote: post #102
alexpotato wrote: post #2

I have always loved SQLite. I have also heard that some firms ban its use. Why? Because it makes it SO easy to set up a database for your app that you end up with a super critical component of your application that looks exactly like a file. A file that can have any extension. And that file can be copied around to other servers. Even if there is PII in that file. Multiply this times the number of applications in your…

I recently watched a YT video about this subject: https://www.youtube.com/watch?v=lSVgeMoXJTs In summary, companies use the bus-metric to see how viable a project is. Bus, as in, how many people can be hit by a bus before there is no one left to maintain the project. Despite its ubiquity, SQLite is maintained by only 3 people. That bus-metric for SQLite is 3, which is way too low for some companies. Give the link a w…

and anyone that considers this to be the case for sqlite, should probably have their reasoning skills examined.

if the unfortunate bus incident happens to sqlite developers, there is exactly ZERO chance that it will not be very well maintained on the count of all the users, many of whom already have support contracts going for decades, and which would require the same level of support they have already enjoyed.

Re: SQLite Is a Library of Congress Recommended Storage Format

#199

Post by garaetjjte »

StilesCrisis wrote: post #117
iknowstuff wrote: post #66

Earlier quoted context omitted.

> Why else would MS not support BTRFS/ZFS/Ext or whatever? You seriously can’t think of another reason? File systems are complex. Maintenance is a huge burden. Getting them wrong is a liability. Reason enough to only support the bare minimum. And then, 99% of their users don’t care about any of those. NTFS is good enough

NTFS is dog slow. Unfortunately it's nowhere near good enough.

I don't think this is necessarily fault of NTFS, but Windows filesystem stack.

Re: SQLite Is a Library of Congress Recommended Storage Format

#200

Post by pletnes »

perching_aix wrote: post #89
pletnes wrote: post #44

Earlier quoted context omitted.

You can unzip the xlsx and read the xml inside. It’s not the worst format by far.

What would you reckon is the worst format? I'm very curious of your standards given this.

A binary format that is only readable by some very specific version of the program writing it. The older xls comes to mind, but there must be thousands of examples.
Post reply on HN