Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite is all you need for durable workflows

obeli.sk

Post reply on HN
Print view View original

291–300 of 413 posts

Re: SQLite is all you need for durable workflows

#291

Post by jklmnopqrstuvw »

glzone1 wrote: post #246
petcat wrote: post #84

Earlier quoted context omitted.

Woof. That sounds very complicated. If you need that kind of write concurrency, use an unlogged table in postgres [0]. Then you don't have to invent a whole sharded thing yourself. [0] https://www.postgresql.org/docs/current/sql-createtable.html...

Running postgresql is an order of magnitude more complicated than sqlite. 130k tps even with unlogged is not always super easy especially if getting hit concurrently. Postgresql connection overhead alone can be pretty brutal if you are setting up and tearing down connections or have 1,000 writers etc. Postgresql generally requires good network connectivity. Folks doing sqlite distributed tend to have everything indep…

even with wal, postgresql can easily reach 130k tps in pipeline mode.

Re: SQLite is all you need for durable workflows

#292

Post by mxey »

franga2000 wrote: post #282
HDThoreaun wrote: post #268

Earlier quoted context omitted.

Every container gets its own database?

Yes? Well, every "app", as I quite explicity wrote. Look up the docker compose file or helm chart for basically any app. I'm running dozens of apps, each with their own postgres, redis and nginx containers alongside the main application server. That's what the stack is designed for.

The Compose file is written like that so you can quickly try the app without setting up extra dependencies. Usually not for production use.

Especially since in production you might want to scale the parts separately. I like to have a Postgres cluster to connect where backup is already handled, and the app then doesn’t have any persistent data, doesn’t need any network volume mounts.

Re: SQLite is all you need for durable workflows

#293

Post by formerly_proven »

pseudalopex wrote: post #203
formerly_proven wrote: post #144

Earlier quoted context omitted.

This is the fault/price of backwards compatibility. Most users of SQLite should just fire off a few pragmas on each connection: PRAGMA journal_mode = WAL PRAGMA foreign_keys = ON # Something non-null PRAGMA busy_timeout = 1000 # This is fine for most applications, but see the manual PRAGMA synchronous = NORMAL # If you use it as a file format PRAGMA trusted_schema = OFF You might need additional options, depending on…

The price of compatibility could be a pragma.

It literally is? Changing the defaults shown in the PRAGMAs above would break backwards compatibility. SQLite is strictly semantically versioned and does not break backwards compatibility.

https://sqlite.org/versionnumbers.html

Re: SQLite is all you need for durable workflows

#294

Post by mxey »

levkk wrote: post #22

I don't understand this obsession with SQLite for real, production apps. SQLite is an embedded database, completely unsuitable for managing concurrency. This is what database _servers_ are for, e.g., Postgres, MySQL, etc. Their entire job is to allow you to modify data from multiple processes, on different machines, at the same time. This is a foundational principle of computer science. It seems to me that the "SQLit…

I think the SQLite website itself says it best:

> SQLite does not compete with client/server databases. SQLite competes with fopen().

Re: SQLite is all you need for durable workflows

#295

Post by password4321 »

stephenlf wrote: post #64

Can’t wait to see the next iteration of this idea with “Logs are all you need for durable workflows.”

Pardon my ignorance trying to follow up on what is most likely sarcasm but is this not Kafka's claim to fame?

I am joining a new project and need to know to what extent Kafka is still a part of the future for new big data projects. It doesn't seem like there are alternatives at the high end but instead the question is when other technologies (that are easier to manage, require less compute, etc.) max out.

Re: SQLite is all you need for durable workflows

#296

Post by bambax »

mxey wrote: post #288
bambax wrote: post #286

Earlier quoted context omitted.

> SQLite is an embedded database Yes, but that's not its main selling point. An SQLite database is also a single file, which makes it incredibly easy to replicate, backup, transfer, restore, etc.

SQLite in WAL mode which you want for server apps is multiple files. Files which you cannot just copy while your application is running if you want a correct backup.

Vacuum into or .backup work perfectly with a running, WAL enabled db.

Re: SQLite is all you need for durable workflows

#297

Post by freakynit »

SQLite backed with Raid-10 NVME disks and periodic backups to cloud storage is generally more than enough to run majority of the production workloads of startups.

Writes are single threaded, but, you can still easily do thousands per second.

DuckDB offers similar qualities, on the OLAP side.

This is not to say this is best combination.. but, when you consider the simplicity of setup, usage, operations, and backups, and cost element, this indeed offers one of the best, if not the best combination.

Re: SQLite is all you need for durable workflows

#298

Post by mxey »

bambax wrote: post #296
mxey wrote: post #288

Earlier quoted context omitted.

SQLite in WAL mode which you want for server apps is multiple files. Files which you cannot just copy while your application is running if you want a correct backup.

Vacuum into or .backup work perfectly with a running, WAL enabled db.

At which point there’s little difference from any other database’s backup commands.

Re: SQLite is all you need for durable workflows

#299

Post by password4321 »

bitexploder wrote: post #7

I started setting up my workflows using Temporal. It deploys as relatively light weight local app. For an isolated local installation it uses SQLite. It makes the process of dealing with API retries and organizing workflows and tasks really simple. I recommend giving it a try. It is, philosophically, exactly what this article is suggesting, but it adds an incredibly rich and flexible interface for agents to work with…

> It saves a lot of tokens when an agent can just query a specific row instead of having to fire up jq or grep through markdown

Just wanted to make sure no one missed this point in your comment because eventually users will be paying the full cost for tokens instead of VC's paying, with GitHub Copilot's pricing realignment leading the way.

Re: SQLite is all you need for durable workflows

#300

Post by bambax »

mxey wrote: post #298
bambax wrote: post #296

Earlier quoted context omitted.

Vacuum into or .backup work perfectly with a running, WAL enabled db.

At which point there’s little difference from any other database’s backup commands.

I don't think you can install "any other database" by pasting one file in a direcory somewhere? Even if you can produce such a backup with the same command.
Post reply on HN