Live data from Hacker News

SQLite is all you need for durable workflows

obeli.sk

211–220 of 413 posts

Re: SQLite is all you need for durable workflows

#212
post #74

Earlier quoted context omitted.

Imagine if every tweet had to go through a one-at-a-time queue before being persisted. There's about 6000 tweets per second, so you would have to be able to save them at <0.17ms per tweet or else you would become backlogged. If you are getting backlogged, you have to buffer those incoming tweets somewhere until they can be writted, and eventually that buffer gets full and you start losing tweets.

Sqlite can quite easily do 5000+ insert+commits per second on typical NVMe drives. Speed is rarely the constraint that makes it unsuitable for an application.

Round trip time is actually much faster than Postgres, since there’s no need to touch the network. You can get massive single threaded throughput. In order to achieve comparable throughput in Postgres you need a large amount of concurrent connections, since each conn spends most of its time passing messages, deserializing etc (with a much larger total amount of overhead). There are a surprising amount of bottlenecks and misconfiguration that can tank performance of networked systems, particularly DBs.

Like you suggest, the reason for not picking SQLite is not reliability, speed, etc. Networked DBs allow decoupling between app and db servers, which have operationally different characteristics. But most importantly, you can have multiple apps access the same DB at the same time. Eg analytics, one off queries, any 3p app that interacts with your data directly.

Re: SQLite is all you need for durable workflows

#214
I've replaced all of these with Go + SQLite:

1. Intercom 2. Zendesk 3. Email marketing 4. Kanban 5. Todo 6. Our billing stack 7. Our issue tracker 8. Our forum 9. Uptime monitor 10. PagerDuty (clone)

I have dozens of products I sell, so I thought: why not build everything ourselves?

All of these run on the same server and use very little memory. I replaced all the SaaS tools we used with these.

I also moved to dedicated servers and dropped costs to about 1/10th of what we were paying for managed cloud solutions, while maintaining the same HA and even achieving lower latency (partly because noisy neighbors on VPSes were increasing tail latency).

We used to spend a ton on this stuff. These have now been in production for four months and have only needed minor updates.

Deployment is dirt simple. No Docker, no Kubernetes—just a systemd service and a binary built on the dev machine and deployed.

We also used to pay for services like MaxMind and IPData. I ended up hand-rolling my own IP geolocation service, which, in my tests, outperforms most existing solutions.

It all started with replacing Uptime Robot. Then I got more confident and replaced PagerDuty. After that, I replaced Intercom.

Finally, I had always heard people say, "Don't build your own billing stack." But I said YOLO, let me make that mistake myself. So I studied our existing billing solution, developed my own, and rolled it out. So far, we've had zero issues with it.

Caddy in front.

I found that we only use maybe 1–5% of the features most SaaS products offer, while the features we actually need keep getting buried deeper and deeper inside these "enterprise-grade" platforms, making our workflows more difficult.

I won't show my commercial products because our partners and clients probably wouldn't appreciate knowing how cheap I am—but I call it being resourceful.

I can show my free app, though, which has 20,000+ users and was launched recently: https://macrocodex.app/

It only uses the Zendesk clone. Email is handled through Cloudflare routing, so we pay almost nothing to run the app.

Re: SQLite is all you need for durable workflows

#215

Can’t wait to see the next iteration of this idea with “Logs are all you need for durable workflows.”

Wait no further. It's already happening.

One reason why a "logs are all you need" solution may fail: untrusted-log-as-injection[1].

Check those SBOM, and don't forget to include their CICD pipelines[2].

[1] https://news.ycombinator.com/item?id=48315440

[2] https://github.com/jqwik-team/jqwik/issues/708#issuecomment-...

Re: SQLite is all you need for durable workflows

#216

I've replaced all of these with Go + SQLite: 1. Intercom 2. Zendesk 3. Email marketing 4. Kanban 5. Todo 6. Our billing stack 7. Our issue tracker 8. Our forum 9. Uptime monitor 10. PagerDuty (clone) I have dozens of products I sell, so I thought: why not build everything ourselves? All of these run on the same server and use very little memory. I replaced all the SaaS tools we used with these. I also moved to dedica…

I rolled my own uptime but how do you vary region or do residential testing ?

Re: SQLite is all you need for durable workflows

#218

I've replaced all of these with Go + SQLite: 1. Intercom 2. Zendesk 3. Email marketing 4. Kanban 5. Todo 6. Our billing stack 7. Our issue tracker 8. Our forum 9. Uptime monitor 10. PagerDuty (clone) I have dozens of products I sell, so I thought: why not build everything ourselves? All of these run on the same server and use very little memory. I replaced all the SaaS tools we used with these. I also moved to dedica…

I rolled my own uptime but how do you vary region or do residential testing ?

I actually don't.

I just have uptime service hosted outside of our main infra. It connects to my service called Siren, which alerts me on my phone with an alarm on full volume with SWAT cat intro.

It's good enough for what we do, barely have any downtime. But it helped me figure out 6s downtime we would experience when our spot instances get knocked out, so it helped me increase health check frequency

6s downtime is a lot when you are getting hammered at 100 RPS.

Re: SQLite is all you need for durable workflows

#220
post #3

> The caveat is that Litestream replication is asynchronous. A restore can miss the newest local writes if the SQLite volume disappears before they are copied. That is fine for many AI and experimentation workflows In short: SQLite is not all you need, unless you’re just experimenting don’t actually care about durability, in which case you also need litestream + object storage. Right.

try setting up replication/failover in postgres, it's much more work.
Post reply on HN