Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite Is a Library of Congress Recommended Storage Format

sqlite.org

Post reply on HN
Print view View original

131–140 of 205 posts

Re: SQLite Is a Library of Congress Recommended Storage Format

#131

Post by tnelsond4 »

chungy wrote: post #118
raxxorraxor wrote: post #100

Earlier quoted context omitted.

SQLite is simple in its own way and I like the design principle of their SQL dialect. "Right joins are just left joins in the wrong direction, you don't need that crap" Of course it always gets simpler or more specialised. I think many apps using databases would run with SQLite just as well. And some would probably run just as well with a textfile instead of any db like SQLite.

> "Right joins are just left joins in the wrong direction, you don't need that crap" SQLite has supported all types of joins since version 3.39 in 2022.

I must've messed something up, but I remember some joins (was it full outer join?) being unbelievably slow? Was I doing something wrong?

Re: SQLite Is a Library of Congress Recommended Storage Format

#132

Post by tnelsond4 »

gpvos wrote: post #82
tnelsond4 wrote: post #33

I'm always inspired by SQLite. Overall I like it, but if you're not doing writes it's really overkill. So I made a format that will never surpass SQLite, except that it's extremely lighter and faster and works on zstd compressed files. It has really small indexes and can contain binaries or text just like SQLite. The wasm part that decompresses and reads and searches the databases is only 38kb (uncompressed (maybe 16…

A more standard solution would be cdb.[0] Although that doesn't support compressed data. [0] https://cdb.cr.yp.to/ , https://en.wikipedia.org/wiki/Cdb_(software)

I'll definitely have to look at this. I'm using binary search right now because it's fast enough and I understand it better, but I might replace it with hashes at some point.

Re: SQLite Is a Library of Congress Recommended Storage Format

#133

Post by LoganDark »

lenwood wrote: post #127

Just yesterday it occurred to me that it had been a while since I last saw an SQLite post at the top of HN. I really like the simplicity and speed of SQLite, I've used in both personal and professional projects. For day-to-day work I still end up in Excel, not because I like it more (I don't), but because its ubiquity makes it the lowest friction way to share & explore datasets with less technical stakeholders and ex…

I've always been irked by how SQLite relies on text parsing to work. Why is it that I have to write queries in text rather than expressing them in programmatic logic? I have never used a relational database because of this, because I hate them, but they can be more performant than pure structured data, but I hate SQL and the entire idea of SQL and I don't want to write it or learn it or use a system that relies on it. It feels like the wrong approach, on the level of PHP. Is there anything I can do to help this? I don't want to keep passing up SQLite just because of SQL, but I just can't seem to agree with it. I don't want to build strings or have string parsing anywhere in the stack, it just feels wrong.

Re: SQLite Is a Library of Congress Recommended Storage Format

#134

Post by gcardinal »

lenwood wrote: post #127

Just yesterday it occurred to me that it had been a while since I last saw an SQLite post at the top of HN. I really like the simplicity and speed of SQLite, I've used in both personal and professional projects. For day-to-day work I still end up in Excel, not because I like it more (I don't), but because its ubiquity makes it the lowest friction way to share & explore datasets with less technical stakeholders and ex…

I'm under no illusion I'll suddenly shatter your world views with this, but in case it's as useful to you as it was to me, you might want to check out Metabase[1].

You can self-host and if all you care about is showing data in a digestible format to stakeholders, it's really simple. You can of course go overboard and regret all of your life's decisions with it, but I try and abstain myself.

[1]: https://www.metabase.com/

Re: SQLite Is a Library of Congress Recommended Storage Format

#135

Post by forinti »

Fwirt wrote: post #5
alexpotato wrote: post #2

I have always loved SQLite. I have also heard that some firms ban its use. Why? Because it makes it SO easy to set up a database for your app that you end up with a super critical component of your application that looks exactly like a file. A file that can have any extension. And that file can be copied around to other servers. Even if there is PII in that file. Multiply this times the number of applications in your…

The question is, do the same firms ban Excel? Excel spreadsheets often end up as shadow databases in unlikely places.

Don't get me started on Access...

Re: SQLite Is a Library of Congress Recommended Storage Format

#136

Post by irishcoffee »

perching_aix wrote: post #96
irishcoffee wrote: post #94

Earlier quoted context omitted.

3rd order effect, people copy and paste the old sheet into a new sheet, now we have worse exponential. You’re not very good at this huh.

Which is very annoying and people will complain. People complaining can be then directed towards a better solution. As a bonus, mistakes will also rise, leading to further complaints, especially ones that reach higher. All this making the dogshit practice, and the idiots committing them, infinitely more visible and thus fixable. The sheer volume of data that needs tending to may even grind certain departments to a ha…

Sorry for the snark, that was shitty of me.

Re: SQLite Is a Library of Congress Recommended Storage Format

#137

Post by [deleted] »

xxs wrote: post #39
maxloh wrote: post #32

Earlier quoted context omitted.

FYI, they added a lot more formats to the list after that. Preferred 1. Platform-independent, character-based formats are preferred over native or binary formats as long as data is complete, and retains full detail and precision. Preferred formats include well-developed, widely adopted, de facto marketplace standards, e.g. a. Formats using well known schemas with public validation tool available b. Line-oriented, e.g…

.7z being there just discredits the entire process. The underlying compression algorithm is a free-hand one and can be anything[0], or contain bugs and exploits[1]. Personally I use only zstd with .7z which is 'non-standard' by the official (Russian) release. [0]: https://7-zip.org/7z.html [1]: CVE-2025-0411

[deleted]

Re: SQLite Is a Library of Congress Recommended Storage Format

#138

Post by jodrellblank »

pbhjpbhj wrote: post #63
tombert wrote: post #43

Earlier quoted context omitted.

Everywhere exFAT is supported now. Windows, Mac, Linux, FreeBSD would be fine.

Presumably Microsoft fear making it easy to swap OSes and access the same data. "I can use Linux because if I get stuck I can just switch to Windows and still access my data" is a comfort that probably keeps people from even trying Linux (or other OSes)? Why else would MS not support BTRFS/ZFS/Ext or whatever? {I'm not saying that I think this works.}

Have you seen Linus Torvalds' comments on ZFS from 2020?

https://www.realworldtech.com/forum/?threadid=189711&curpost...

".. there is no way I can merge any of the ZFS efforts until I get an official letter from Oracle that is signed by their main legal counsel or preferably by Larry Ellison himself .. Don't use ZFS. It's that simple. It was always more of a buzzword than anything else, I feel, and the licensing issues just make it a non-starter for me. .. The benchmarks I've seen do not make ZFS look all that great. And as far as I can tell, it has no real maintenance behind it either any more, so from a long-term stability standpoint, why would you ever want to use it in the first place?"

BTRFS: RedHat has removed all support for BTRFS and deprecated it: https://access.redhat.com/solutions/197643

BTRFS, "Linux's perpetually half-finished filesystem" by ArsTechnica: https://arstechnica.com/gadgets/2021/09/examining-btrfs-linu... with many problems still unaddressed in 2021 dating back to 2009.

Re: SQLite Is a Library of Congress Recommended Storage Format

#139

Post by euroderf »

hermitShell wrote: post #6
Fwirt wrote: post #5

Earlier quoted context omitted.

The question is, do the same firms ban Excel? Excel spreadsheets often end up as shadow databases in unlikely places.

The sane thing would be to ban Excel and promote SQLite. Excel is often used for tabulated text (issue tracking) not calculations. Perfect use case for a relational db

Well heck can't someone make an SQLite extension that is basically just a simplified Excel ?

Re: SQLite Is a Library of Congress Recommended Storage Format

#140

Post by mcdonje »

Fwirt wrote: post #5
alexpotato wrote: post #2

I have always loved SQLite. I have also heard that some firms ban its use. Why? Because it makes it SO easy to set up a database for your app that you end up with a super critical component of your application that looks exactly like a file. A file that can have any extension. And that file can be copied around to other servers. Even if there is PII in that file. Multiply this times the number of applications in your…

The question is, do the same firms ban Excel? Excel spreadsheets often end up as shadow databases in unlikely places.

PII sniffers are pretty good at dealing with excel files. Excel is seen more as an analyst tool than a dev tool. Any place that bans Excel needs to either let analysts use some other turing complete data tools, like python or R or something, or they'll have trouble attracting analyst talent. They'll have devs and data entry users and that's it.

The only way that works is if the dev team is large enough to be responsive to business needs, which almost never happens because devs are expensive. The juniors who are tweaking business logic every day are functionally doing a role analysts can do if you just give them a sane API and data tools.

Post reply on HN