Live data from Hacker News

SQLite is all you need for durable workflows

obeli.sk

391–400 of 413 posts

Re: SQLite is all you need for durable workflows

#391
post #94

Earlier quoted context omitted.

Sqlite is good for lots of stuff, but you're probably focusing your days on high-scale webapps that want sharding with networked DBs. That's one domain, and an interesting one, but there are lots of others. I'm a big fan of re-evaluating prior "best practices" in light of technology changes, especially in ways that improve simplicity. Running my family's social media site off a single sqlite DB on a VPS is great. ~15…

> Running my family's social media site off a single sqlite DB on a VPS is great. ~15 users, almost zero maintenance. Details, please!

Nothing public at the moment, unfortunately. I was kind of surprised at the lack of Very Simple scripts to just host a site for a handful of users easily. So I wrote one that focuses on:

    Unlimited-length posts in a chronological feed. You don't have to subscribe to everyone - having something appear in your feed is opt-in.
    Circles. Subscribing to someone into a hobby can get noisy. Circles give the hobby a place, for those that want to check.
    RSS everywhere: Anyone can add an RSS feed to the server, and anyone can view all the subscribed feeds and choose which to follow. They are not part of the home feed, but a separate section. Every feed (circle, user) has a public-to-the-internet feed and a private-to-folks-on-the-server feed.
    Mentions: you can @-mention anyone on the server to get a post into their home feed.
    Public posts: by default, every post is private. If a user checks a box, the post will be made public, so folks that are not logged in can see it.
    Posts in the feeds section, circles section, or home section can be replied to, with the user choosing where to share (a circle, or their feed)
    It's all a single-file Python script that can run in either CGI mode or server mode. I compile it with nuitka and run it as a CGI behind Apache. Very old school, works fine. Non-attachment data stored in sqlite, so if you have the DB, you can fire up a copy of the site, sans attachments.
    Attachments: gallery posts supported, with lightbox viewing. PDFs supported. Nothing else right now.
    Works on mobile.
    No email or other form of notifications. If users visit, they see stuff. If they don't, they don't.
    Super-opinionated: admin controls everything, and password resets go through the admin, who simply asks the server to regenerate a new password, that the admin then passes along to the user.
    There are no direct messages, or private posts, in the sense that if you log in, you'll be able to see everything going on if you click through to it.
    Replies, comments, and reactions are supported. Conversation view (tree of posts replying to each other) is supported.
    
Those are the features off the top of my head. It's a social network for small groups that are high-trust. If I open source it (after I feel it is more airtight) I'll probably ask AI to provide a landing page using this as a prompt and provide this verbatim at the top of that page for folks that want the zero-bullshit version. =)

Re: SQLite is all you need for durable workflows

#392

I started setting up my workflows using Temporal. It deploys as relatively light weight local app. For an isolated local installation it uses SQLite. It makes the process of dealing with API retries and organizing workflows and tasks really simple. I recommend giving it a try. It is, philosophically, exactly what this article is suggesting, but it adds an incredibly rich and flexible interface for agents to work with…

What's wrong with looking up the algorithm called "work stealing" and just implementing it? Making web UIs for a working algorithm is something AI is really, really good at (and I'll actually trust it with that)

Re: SQLite is all you need for durable workflows

#394

Earlier quoted context omitted.

How do you do server maintenance or handle hardware failure if your database is SQLite? You are going to have to take downtime, even in the best scenario.

1. Proxmox live migration or HA, Ceph storage 2. K8S DaemonSet, PVC backed by probably Ceph 3. Just..don't care? Do maintenance outside of working hours, fix issues quickly and explain things nicely to your customers. Not everything is google-scale. Most people can deal with some downtime. And it's not like you won't have downtime in let's say a postgres-backed app. But now you have two "servers" to deal with.

Those first two options add more complexity than just using an external database.

I guess I am just not used to downtime being acceptable. Spent most of my career working for a CDN, any sort of downtime was simply unacceptable. I can't stop myself from that sort of thinking now.

Re: SQLite is all you need for durable workflows

#396
post #196

Earlier quoted context omitted.

there is a difference between concurrency in a distributed environment and concurrency on a single machine across processes. SQLite is incredibly useful for the latter. you seem like the inexperienced one to me..

SQLite does not support concurrent writes at all (on a single machine), a single writer process locks the entire database.

not really true, SQLite supports WAL mode which allows concurrent writes (technically write _attempts_, but these writes are exceptionally fast and are serialized to the file-system anyway, so functionally equivalent to concurrent writes for p50 use case).

also, use-case for massively concurrent writes is pretty narrow, and SQLite is not optimizing for that anyway.

Re: SQLite is all you need for durable workflows

#397

Earlier quoted context omitted.

1. Proxmox live migration or HA, Ceph storage 2. K8S DaemonSet, PVC backed by probably Ceph 3. Just..don't care? Do maintenance outside of working hours, fix issues quickly and explain things nicely to your customers. Not everything is google-scale. Most people can deal with some downtime. And it's not like you won't have downtime in let's say a postgres-backed app. But now you have two "servers" to deal with.

Those first two options add more complexity than just using an external database. I guess I am just not used to downtime being acceptable. Spent most of my career working for a CDN, any sort of downtime was simply unacceptable. I can't stop myself from that sort of thinking now.

It's complexity you already have. You need some sort of HA for your app server and some sort of resilient storage for your database server. Using sqlite just means the storage is used by the app server directly, nothing more.

Re: SQLite is all you need for durable workflows

#399
The thing that people forget about databases is that they are just really complicated ways of writing to a file.

The crux of all these “you only need sqlite” posts isn’t that you need SQLite, it’s that your architecture only ever needed flat appendable log file to begin with.

Once you realise that SQLite is just an aggregated view over this log it calls into question if you needed a third party query engine at all, suddenly the whole NoSQL vs SQL debate becomes a meaningless implementation detail.

Server based databases aren’t special beyond the fact they’re just distributed synchronisation primitives over files on a file system. But the utility of this is quite limited in practice, writing to a table will always require some sort of synchronisation, using SQLite vs Postgres is just a case of where you put the lock - in your own code or in a 3rd party server.

Re: SQLite is all you need for durable workflows

#400
post #107

Earlier quoted context omitted.

> SQLite can work well for the parts of your system where there is naturally strong partitioning. Or the parts of your system that don't have big data and no need for massively concurrent writes. And that's the vast majority of systems!

What's massive amount concurrent writes? What's big data?

Such an exercise is left for the reader.
Post reply on HN