Live data from Hacker News

Learning a few things about running SQLite

jvns.ca

71–80 of 101 posts

Re: Learning a few things about running SQLite

#71
post #60
post #59

Earlier quoted context omitted.

does write only prevent overwrites?

Good point, no it doesn't.

but i suppose if you're prefixing your entries with enough entropy, it'd be quite difficult for adversaries to predict and overwrite files without an `ls` equivalent

Re: Learning a few things about running SQLite

#72
post #67
post #33

Earlier quoted context omitted.

Looks similar to EXPLAIN QUERY PLAN: https://sqlite.org/eqp.html Raw EXPLAIN dumps bytecode, which is usually much more verbose than you want. EXPLAIN QUERY PLAN dumps a summary.

In my experience, SQLite explain plans are by far the most useless of any database out there. No concept of costing, no buffer information, no explain analyze. It's almost like they don't want you looking at it.

sqlite competes with fopen. If you don't like it, use a real database.

Re: Learning a few things about running SQLite

#73
post #2

> I’ve been backing up to AWS, which is always a pain because it’s annoying to navigate the AWS console to generate credentials. I got so annoyed with that a few years ago that I ended up building a whole tool just to solve that one problem: uvx s3-credentials create my-existing-s3-bucket This spits out read-write credentials that are scoped JUST for that bucket. You can add --read-only or --write-only to have creden…

A more general solution for dealing with complex AWS services is learning just enough terraform to let LLMs do the rest. It also makes it much easier to tear stuff down later as you won’t need to remember what you created.

Re: Learning a few things about running SQLite

#74

Earlier quoted context omitted.

Once you release software to the world, the world can choose use it however it wants. Still, the author's of SQLite document their intention for it to be used locally on their "when to use" page. > SQLite strives to provide local data storage for individual applications and devices. https://www.sqlite.org/whentouse.html

To be fair they also say > Generally speaking, any site that gets fewer than 100K hits/day should work fine with SQLite.

I have a 100GB SQLite DB in use that gets 8+ million rows added dally. It's on an off-the-shelf nvme SSD in a "server" that I build from 5+ year old parts.

Those are writes not hits so not directly comparable.

Re: Learning a few things about running SQLite

#75
post #37

Earlier quoted context omitted.

So about one per second (up to ten, less conservatively). I concur. But if you think your site might ever scale beyond that, do yourself a favor and use Postgres from the get-go.

They also say it seems to work well up to 500k a day, which is quite a bit.

I have 8+ million rows added daily to a 100+GB DB. There is a limit somewhere but I haven't found it yet.

Re: Learning a few things about running SQLite

#76

> I didn’t care to investigate further and > my best guess and > and presumably other things?) and > maybe there’s a bunch of Python code running inside a transaction Basically, this article has no substance. The author didn't bother to learn anything, didn't look things up. And is then wildly guessing, sometimes wrong. This is BTW the reason why (as a Debian user) if I search something Linux related and a Ubuntu for…

Don’t tell me you you found a load bearing seam. Honest take, the blast radius of that footgun muestra have read impressive before it was posted

Re: Learning a few things about running SQLite

#77

> I didn’t care to investigate further and > my best guess and > and presumably other things?) and > maybe there’s a bunch of Python code running inside a transaction Basically, this article has no substance. The author didn't bother to learn anything, didn't look things up. And is then wildly guessing, sometimes wrong. This is BTW the reason why (as a Debian user) if I search something Linux related and a Ubuntu for…

[dead]

Re: Learning a few things about running SQLite

#78

> I didn’t care to investigate further and > my best guess and > and presumably other things?) and > maybe there’s a bunch of Python code running inside a transaction Basically, this article has no substance. The author didn't bother to learn anything, didn't look things up. And is then wildly guessing, sometimes wrong. This is BTW the reason why (as a Debian user) if I search something Linux related and a Ubuntu for…

I actually think the article is great - it shows the experience of someone who is a good approximation of a smart user that uses the tech. The posters focus is clearly on running the website, and these are the sorts of things that trip up everyday users that don’t spend their day in these tools.

Off the top of my head, yesterday in work I used 2 programming languages, 2 build systems, a cloud provider, a secrets manager, a very intricate framework for client server communication in both languages, plus my VCS, editor and CI tool. That’s a fairly typical set of tools to use for one feature for me, before you go into the weeds of OS versions, specific runtime versions, databases, reverse proxies, caches, and the domain logic!

If I went deep on every single thread exposed to me, I’d never get anything done, so I have to choose my battles just like OP has done here

Re: Learning a few things about running SQLite

#79

IMHO for a small DB I’d encourage sending out an email on each successful backup to ensure it’s completed successfully as a safety check, and zipping it up and emailing it to a known account even. With inboxes being able to take gigabytes, it’s a no brainer. This can be done daily or weekly. And yes, never allow the files to be deleted from outside. The transfer is a one way valve. If uploading, it’s a write-only ope…

Nah, write only upload to S3/R2 is the way to go. Why would you add all of the headaches of sending email to your tiny backup.

It’s as much work, and scales 1000x farther than the email approach.

Re: Learning a few things about running SQLite

#80
post #74

Earlier quoted context omitted.

To be fair they also say > Generally speaking, any site that gets fewer than 100K hits/day should work fine with SQLite.

I have a 100GB SQLite DB in use that gets 8+ million rows added dally. It's on an off-the-shelf nvme SSD in a "server" that I build from 5+ year old parts. Those are writes not hits so not directly comparable.

So many problems in tech would be solved with $250 spent on an NVMe drive.
Post reply on HN