Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite is all you need for durable workflows

obeli.sk

Post reply on HN
Print view View original

81–90 of 413 posts

Re: SQLite is all you need for durable workflows

#81

Post by m2f2 »

There's a wide gap from files to multipartition databases. Running databases in a container is not for me sorry whenever real production stuff is on the table.

Personally, lots of ETL can just be taken care of locally without involving enterprise databases. In such cases, DuckDB is 5x-10x better than SQLite and orders of magnitude simpler/faster than spinning up a dedicated Postgres database.

For general scripting, there's no match between a 20-lines awk script and a much cleaner, robust, maintainable equivalent SQL script based on DuckDB.

I just hope MotherDuck don't need to pump/dump for IPO - it would be sad losing that tool for the usual corporate greed.

Re: SQLite is all you need for durable workflows

#82

Post by this_user »

stephenlf wrote: post #64

Can’t wait to see the next iteration of this idea with “Logs are all you need for durable workflows.”

Shortly followed by:

"Sockets are all you need for durable workflows" and then finally "Kernel primitives are all you need for durable workflows."

But seriously, part of being a professional is using the right tool for the job.

Re: SQLite is all you need for durable workflows

#84

Post by petcat »

lanstin wrote: post #53
levkk wrote: post #22

I don't understand this obsession with SQLite for real, production apps. SQLite is an embedded database, completely unsuitable for managing concurrency. This is what database _servers_ are for, e.g., Postgres, MySQL, etc. Their entire job is to allow you to modify data from multiple processes, on different machines, at the same time. This is a foundational principle of computer science. It seems to me that the "SQLit…

I had very good results giving 1 SQL DB per go routine, so the accesses were serialized up front, on a very high volume (130K requests/second) service. Exact transactionality was not a product goal, and the SQLite was just to backup the in memory state. If we lost a little due to abend or something, that was ok (although for normal maintenance it caught SIGTERM and stopped the listen and then waited for in flight cal…

Woof. That sounds very complicated. If you need that kind of write concurrency, use an unlogged table in postgres [0]. Then you don't have to invent a whole sharded thing yourself.

[0] https://www.postgresql.org/docs/current/sql-createtable.html...

Re: SQLite is all you need for durable workflows

#85

Post by eterm »

teaearlgraycold wrote: post #23
levkk wrote: post #22

I don't understand this obsession with SQLite for real, production apps. SQLite is an embedded database, completely unsuitable for managing concurrency. This is what database _servers_ are for, e.g., Postgres, MySQL, etc. Their entire job is to allow you to modify data from multiple processes, on different machines, at the same time. This is a foundational principle of computer science. It seems to me that the "SQLit…

Well if you run a tiny single-threaded app then SQLite is a nice simplification over spinning up a separate machine for Postgres.

Or you can run postgres on the same machine as the application, which lets you much more easily migrate if the time comes when you need to scale to multiple application servers.

There's a world between "local file" and "network DB server", running a DB server locally has lots of benefits from being able to easily query from outside if needed to forcing you to consider concurrency without the latency overhead of a network hop.

Re: SQLite is all you need for durable workflows

#86

Post by sevenzero »

strbean wrote: post #74
sevenzero wrote: post #44

Earlier quoted context omitted.

I guess I need to dive deeper into this as I do not understand the implications you gave me, but I appreciate the attempt. Generally I understand why concurrency is good in many cases, I just dont get why its important for database stuff too. Edit: thanks for clarifying in the edit, makes a lot more sense.

Imagine if every tweet had to go through a one-at-a-time queue before being persisted. There's about 6000 tweets per second, so you would have to be able to save them at <0.17ms per tweet or else you would become backlogged. If you are getting backlogged, you have to buffer those incoming tweets somewhere until they can be writted, and eventually that buffer gets full and you start losing tweets.

While I understand your point and like the explanation, I gotta make the joke that some Tweets should be lost

Re: SQLite is all you need for durable workflows

#88

Post by ibejoeb »

mr_toad wrote: post #61
peterspath wrote: post #56

Earlier quoted context omitted.

That’s why there are billions of SQLite databases right? SQLite is likely used more than all other database engines combined. Billions and billions of copies of SQLite exist in the wild. SQLite is found in: Every Android device Every iPhone and iOS device Every Mac Every Windows 10/11 installation Every Firefox, Chrome, and Safari web browser Every instance of Skype Every instance of iTunes Every Dropbox client Every…

That’s a comprehensive list of single user devices.

Single-user, a single natural person, doesn't striclty mean single-accessor though. I don't think anyone here is suggesting that sqlite is a viable replacement a for any networked client/server postgresql system, but it is certainly capable of handling more than the most basic 1:1 tasks. Beyond that, the point is that you only need a file, so when you have natural data boundaries, a lot of problems decompose to that single user/single concern paradigm.

Re: SQLite is all you need for durable workflows

#89

Post by fourside »

gchamonlive wrote: post #66
stephenlf wrote: post #64

Can’t wait to see the next iteration of this idea with “Logs are all you need for durable workflows.”

Are logs all you need for durable workflows? I'm confused here. How'd persist and query nested or related data over logs? By logs I assume you mean something like elasticsearch or meilisearch?

I read the parents comment as sarcasm and not a serious suggestion.

Re: SQLite is all you need for durable workflows

#90

Post by notawhitemale »

wolttam wrote: post #78
gchamonlive wrote: post #66

Earlier quoted context omitted.

Are logs all you need for durable workflows? I'm confused here. How'd persist and query nested or related data over logs? By logs I assume you mean something like elasticsearch or meilisearch?

Pretty much every durable system has an intent log of some sort. The log provides durability, the database system just integrates that log into a more queryable format.

[dead]
Post reply on HN