Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite Is a Library of Congress Recommended Storage Format

sqlite.org

Post reply on HN
Print view View original

141–150 of 205 posts

Re: SQLite Is a Library of Congress Recommended Storage Format

#141

Post by shermantanktop »

IshKebab wrote: post #123
tmountain wrote: post #116

Earlier quoted context omitted.

Overkill in what way exactly? The LOC of the project shouldn't have any bearing on most people's usage of the project. SQLite is one of the well tested and mature projects in the world. What exactly would motivate someone to use PeakSlab instead? What problem are you solving?

Read the comment. He's using it in WASM form and doesn't want users to have to download 1.2MB of SQLite every time they visit the page.

Client caches are a thing, so this is most relevant for cold-start customers. In that case PeakSlab’s download size is an advantage.

Fwiw LocalStorage is a SQLite db on most browsers, with a kv api. It’s be interesting to have the actual API available.

Re: SQLite Is a Library of Congress Recommended Storage Format

#142

Post by shermantanktop »

giza182 wrote: post #47
tnelsond4 wrote: post #33

I'm always inspired by SQLite. Overall I like it, but if you're not doing writes it's really overkill. So I made a format that will never surpass SQLite, except that it's extremely lighter and faster and works on zstd compressed files. It has really small indexes and can contain binaries or text just like SQLite. The wasm part that decompresses and reads and searches the databases is only 38kb (uncompressed (maybe 16…

Perhaps a dumb question, but how do you get data into it if you’re not doing writes

It’s an RODB. Ship the preindexed data blob.

Re: SQLite Is a Library of Congress Recommended Storage Format

#143

Post by electroly »

giza182 wrote: post #47
tnelsond4 wrote: post #33

I'm always inspired by SQLite. Overall I like it, but if you're not doing writes it's really overkill. So I made a format that will never surpass SQLite, except that it's extremely lighter and faster and works on zstd compressed files. It has really small indexes and can contain binaries or text just like SQLite. The wasm part that decompresses and reads and searches the databases is only 38kb (uncompressed (maybe 16…

Perhaps a dumb question, but how do you get data into it if you’re not doing writes

I have a system that builds SQLite databases and uploads them to S3. Once they're in S3, they are never changed. The program that builds the databases only does writes, and the program that queries the databases only does reads. It uses a VFS to query the database in-place with HTTP range requests.

This is indeed not an optimal setup. A more careful design from first principles would not require seeking around the file as much as SQLite does, we'd do a better job on reading exactly the correct range of bytes for a given query since we know ahead of time what the access patterns are, and we could do reads in parallel. With SQLite we have to be very careful about the schema design to ensure it won't have to seek too many times to answer a query. But SQLite was expedient, and I'm confident I'll always be able to read the files. That's less certain for a custom file format.

Re: SQLite Is a Library of Congress Recommended Storage Format

#144

Post by shermantanktop »

altmanaltman wrote: post #53
DeathArrow wrote: post #21

Earlier quoted context omitted.

Do companies ban text files? Text files are used to store data.

Do companies ban brains? Brains are used to store data.

Brains do not exhibit ACID properties…

Re: SQLite Is a Library of Congress Recommended Storage Format

#145

Post by infogulch »

SQLite is remarkably versatile. Just a couple weeks ago an extension to do cross-process queues, streams, pub/sub etc in SQLite was released:

Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite | 327 points | 94 comments | https://news.ycombinator.com/item?id=47874647

Live notifications was one of the big missing pieces to implement whole apps on a sqlite backend, and now there's a decent solution.

Re: SQLite Is a Library of Congress Recommended Storage Format

#146

Post by ashellunts »

faangguyindia wrote: post #23

I went from thinking “SQLite is a toy product, not reliable for real data" to "lets use SQLite for almost everything" SQLite is very good if you can fit into the single writer, multiple readers pattern; you'll never lose data if you use the correct settings, which takes a minute of Google search to figure out. Today, most of my apps are simply go binary + SQLite + systemd service file. I've yet to lose data. Performa…

Do you use multiple backend nodes? If yes, how do you access sqlite files from different nodes?

Re: SQLite Is a Library of Congress Recommended Storage Format

#147

Post by shermantanktop »

mschuster91 wrote: post #58
slopinthebag wrote: post #11

Earlier quoted context omitted.

Same with devops tbh. "Hey everyone, we need to chose the option that involves us the most and provides us the most job security"

Well... eventually the company learns the lesson the hard way, either because a site goes down or gets 0wned. Then everyone will cry about "how this could happen", and the ops people will tell you in response "we warned you that this would happen, here's the receipts, now GTFO".

Preparing to say “I told you so” is a fairly obvious incentive for someone to act like Chicken Little. And of course sometimes they are right, but not always.

Lots of great people in both devops and security. But when teams position themselves as the conscience of the org and the gatekeepers of production, the defensive victim mentality can get pretty strong.

Re: SQLite Is a Library of Congress Recommended Storage Format

#148

Post by weird-eye-issue »

LoganDark wrote: post #133
lenwood wrote: post #127

Just yesterday it occurred to me that it had been a while since I last saw an SQLite post at the top of HN. I really like the simplicity and speed of SQLite, I've used in both personal and professional projects. For day-to-day work I still end up in Excel, not because I like it more (I don't), but because its ubiquity makes it the lowest friction way to share & explore datasets with less technical stakeholders and ex…

I've always been irked by how SQLite relies on text parsing to work. Why is it that I have to write queries in text rather than expressing them in programmatic logic? I have never used a relational database because of this, because I hate them, but they can be more performant than pure structured data, but I hate SQL and the entire idea of SQL and I don't want to write it or learn it or use a system that relies on it…

I bet you really love LLMs

Re: SQLite Is a Library of Congress Recommended Storage Format

#149

Post by mcdonje »

forinti wrote: post #135
Fwirt wrote: post #5

Earlier quoted context omitted.

The question is, do the same firms ban Excel? Excel spreadsheets often end up as shadow databases in unlikely places.

Don't get me started on Access...

Man, Access could've been so good if they just made an app around SQLite. Or since it's Microsoft and they need to do everything their own way, it would've been so good if they made a flat file DB à la SQLite, but with T-SQL (or a subset thereof) instead of JET-SQL.

Increase interoperability. Funnel data people from Excel into real DB technologies.

And if they did more to blur the lines between spreadsheets and databases, and make it seamless to work out of both Excel and Access, add more spreadsheet features to the data views, etc.

Re: SQLite Is a Library of Congress Recommended Storage Format

#150

Post by y-curious »

alexpotato wrote: post #2

I have always loved SQLite. I have also heard that some firms ban its use. Why? Because it makes it SO easy to set up a database for your app that you end up with a super critical component of your application that looks exactly like a file. A file that can have any extension. And that file can be copied around to other servers. Even if there is PII in that file. Multiply this times the number of applications in your…

Required reading for “anything can become a mission critical database” conversations:

https://www.reddit.com/r/sysadmin/comments/eaphr8/a_dropbox_...

Post reply on HN