So is https://github.com/obeli-sk/obelisk the Rust version of https://github.com/temporalio/temporal (Go)? Can you guys add a comparison between them on the site?
Subscribe to this issue: https://github.com/obeli-sk/obelisk/issues/65
SQLite is all you need for durable workflows
401–410 of 413 posts
Re: SQLite is all you need for durable workflows
#402I started setting up my workflows using Temporal. It deploys as relatively light weight local app. For an isolated local installation it uses SQLite. It makes the process of dealing with API retries and organizing workflows and tasks really simple. I recommend giving it a try. It is, philosophically, exactly what this article is suggesting, but it adds an incredibly rich and flexible interface for agents to work with…
It sounds like you’re running this mostly on a single machine? Temporal gets much more complex with scale. Cassandra isn’t fun to manage. Ringpop and TChannel are hard to debug when things go wrong. The SQL backend support doesn’t support horizontally scaled replicas (just single instance) due to consistency requirements. Depending on how your code is written, modifying code baked into workflows becomes complex, as a…
Did I read your comment correctly that Temporal includes Ringpop in their product? Interesting choice to say the least.
Re: SQLite is all you need for durable workflows
#403Big complex data model with ambiguous query patterns? Postgres Small, well defined, data model with known query patterns? Bespoke model There probably is a place for sqlite and my project space so far hasn't yet well-aligned with it.
Probably going to get some winces for this but I do everything with flat files. Maybe my data aren't massive enough, but I mean I can do the relational thing by just having these metadata in some column, and returning rows that contain my desired information in these columns. Even if the file were too big to fit into memory one could just subset chunks of it and chew through. All this can be done with no dependencies…
IMO flat files is a cromulent solution for a subset of the entire database space. "Everything" is too broad for me but different strokes for different folks. Happy dataing.
Re: SQLite is all you need for durable workflows
#404[flagged]
Re: SQLite is all you need for durable workflows
#405Earlier quoted context omitted.
This reads like an advertisement for Temporal :)
I just spent the last two weeks digging into workflow state engines and temporal was one of the candidates. It is a VC backed fork of Cadence. The got 0.3B funding and whatever positive I read about them on the net I take with a big spoon of salt. Just my 2 cents.
I would love to read your analysis somewhere. It's a current interest of mine, both hobby and professional -- and limited to Elixir, Golang and Rust -- and I'm still slowly scoping the landscape.
Recommendations for good engines? Or just thoughts and analyses?
Re: SQLite is all you need for durable workflows
#406The thing that people forget about databases is that they are just really complicated ways of writing to a file. The crux of all these “you only need sqlite” posts isn’t that you need SQLite, it’s that your architecture only ever needed flat appendable log file to begin with. Once you realise that SQLite is just an aggregated view over this log it calls into question if you needed a third party query engine at all, s…
> Once you realise that SQLite is just an aggregated view over this log it calls into question if you needed a third party query engine at all, suddenly the whole NoSQL vs SQL debate becomes a meaningless implementation detail.
You are correct on the premise, however my observation has been that teams who try to be lean and mean and minimize dependencies always start off believing they don't need a query engine... and they always end up needing one, often making things much worse by refusing to just migrate to any database due to sunk cost fallacy.
I fully agree with your argument that it's all just file operations in the end. To me SQLite wins for everything beyond persistence however: the SQL commands, extensions, backups, changesets / patchsets.
Can you, I, and many others write something that has 10% of SQLite feature set that serves us perfectly? Absolutely! But I don't want to, and in business settings nobody will allow you to hand-roll crucial infrastructure software (like databases) unless they're in full crisis mode and you're the mega-expensive consultant brought in to save the day.
Re: SQLite is all you need for durable workflows
#407Earlier quoted context omitted.
We recently just partitioned the data into many SQLite databases and got away with it. It's telemetry data from IoT devices: one device, one database. Backups are an easy rsync job now instead of streaming a multi gigabyte database with compression that take hours. Reporting will just open each database and aggregate multi device data into another database (Duckdb, SQLite or something else, we'll see). Duckdb is not…
Check out Quack for DuckDB.
I saw that but it's a bit too much friction. We'd rather attach a SQLite database and copy the results into that. Also no worries about version changes and compatibility, the types are not too restrictive. But we'll see.
Re: SQLite is all you need for durable workflows
#408Earlier quoted context omitted.
I mean - I agree for the typical multi-user, SaaS webapp. But I don't think that's what these folks are proposing. If they are - yeesh, count me out. If on the other hand they're talking about single-user, software in the small - hell yeah. In fact, I'd also promote DuckDB in this regard (mostly for analytics) - with the power of a single machine these days, you can do a surprising amount and never have to worry abou…
The typical multi-user SaaS webapp doesn't have anywhere near enough users to overwhelm a single SQLite instance. Of the few that do succeed to the point where that's no longer true, a significant fraction can use techniques like sharding to stretch SQLite further.
Yeah no - sharding SQLite? How about if you know you're going to scale like that, build with something appropriate in the first place.
Re: SQLite is all you need for durable workflows
#409Earlier quoted context omitted.
That is the real truth people are voicing when they say Temporal is heavy. They are really saying: Durable, reliable, distributed workloads are hard and it takes effort to manage! And that is true. I know of no systems that make that genuinely easy. It is a hard discipline. Maybe Temporal makes that harder than it should be, but I have no experience there. There are no free lunches in this space. I have no idea how g…
Have you looked into DBOS? Same thesis: durable and reliable workflows are hard to manage -- it just doesn't have to be as hard as Temporal makes it be :)
Have not. For my workflows this was fine. Good to keep in mind thoUgh. I don’t plan to manage a truly distributed system with it. Plus my only reason to do so is professional and we rolled our own system here due to our size solutions like DBOS or Temporal would not work well.
Re: SQLite is all you need for durable workflows
#410Earlier quoted context omitted.
The typical multi-user SaaS webapp doesn't have anywhere near enough users to overwhelm a single SQLite instance. Of the few that do succeed to the point where that's no longer true, a significant fraction can use techniques like sharding to stretch SQLite further.
Yeah no - sharding SQLite? How about if you know you're going to scale like that, build with something appropriate in the first place.
First, you're very likely underestimating how much load SQLite can handle. SQLite is usually write limited, but for smallish writes it can easily handle thousands a second with very trivial optimization, and with more thought can scale to tens or hundreds of thousands of write transactions per second. In some cases, it can actually out perform traditional server based RDMSs because of reduced overhead and because holding locks on network timescales (which will likely happen even for databases with multiple writers, because eventually you have to deal with two transactions needing to write to the same place) is very inefficient.
Second, I think you're overestimating how hard sharding is here. There are plenty of use cases for which sharding isn't just easy to set up, but the natural thing you'd be likely to do even without scale. Things like e.g. a helpdesk SaaS, where each customer/organization has it's own independent data.
Third, a large part of the point is that you are unlikely to know ahead of time you're going to "scale like that". As I already pointed out, most SaaS apps do not end up having many users. For some that's intentional, but for others the reason is that they simply never caught on. For those cases (and they're the vast majority), cosplaying as a much larger app is a complete waste. It's much better to wait until you're successful enough to need to switch and then use the revenue you now have to solve the scaling problem you ran into.
Fourth, as an aside, ironically the other (slightly less) easy way to shard superficially resembles a common way people cosplay as netflix: splitting your data by domain as "microservices" (although there's a good chance they don't need to be independent processes/on independent machines).