31–40 of 45 posts
Earlier quoted context omitted.
Not only is RabbitMQ durable, it has transactions with rollback. The entire set of problems the author describes are down to Reddit using Rabbit like Redis.
RabbitMQ of 15 years ago did not have those features. I have no idea how reddit uses rabbitMQ now, or if they use it at all.
Earlier quoted context omitted.
To be fair, this is talking about Rabbit from 15+ years ago. I have no idea if it's gotten better since.
Is the title supposed to say "(How) I solved a distributed queue problem 15 years ago"? As it stands it makes it sound like you've been working on this problem for 15 years and just solved it.
I'm kind of astonished that something as well-used as RabbitMQ isn't doing much to keep whatever state it has, coherent in some backup/store/database. Or is this just an implementation detail that I'm unaware of?
Earlier quoted context omitted.
Thanks for sharing your insights! You nailed the key tradeoffs of most durable workflow systems. The callback-style programming model is exactly the pain point we aim to solve with DBOS. Instead of forcing you into a custom async runtime, DBOS lets you keep writing normal functions (this is an example in Python): @DBOS.workflow() def do_thing(foo): return bar # You can still call the workflow function like this: resu…
I think your use of Python decorators is a big usability improvement, with the point being that the glue is still there. You mention that in Go there's "a lightweight one-time registration step" but it seems like in addition to calling the registration steps, you also have to use `dbos.CallAsStep()` when calling sub-steps of a workflow, which is almost identical to the temporal Golang SDK which has you call `workflow…
"since Golang doesn't have decorators in the same way Python does, we still have to have code doing the kind of "manual callback" style I mentioned"
That's exactly right, specifically for steps. We considered other ways to wrap the workflow calls (so you don't have to do dbos.RunWorkflow(yourFunction)), but they got in the way of providing compile time type checking.
As Qian said, under the hood the Golang SDK is an embedded orchestration package that just requires Postgres to automate state management.
For example, check the RunWorkflow implementation: https://github.com/dbos-inc/dbos-transact-golang/blob/0afae2...
It does all the durability logic in-line with your code and doesn't rely on an external service.
Thanks for taking the time to share your insights! This was one of the most interesting HN comment I've seen in a while :)
I had an exponential socket growth issue in the late 1990s being the architect and maintainer of a little known ecommerce company that was powering a good portion of the domestic U.S. This volume problem I had equated to drinking from a fire hose that cannot be turned off yet the hose had to somehow be upgraded in motion to handle even more volume. I was young then and greatly inexperienced to truly comprehend the task I had to solve for and as the volume grew ever so slightly daily so too did the response times grow by milliseconds on transactions that few could see but all internally could understand what it meant over time. We managed to acquire the latest multi SMP hardware from Compaq but even this did not solve for the challenges of the increasing volume. I did solve the issue and it involved my third complete rewrite of the entire software stack into what is now recognized as microservices and durable message queueing. Those past choices, to this day, continue to stand the test of time.
Earlier quoted context omitted.
Kafka is a wonderful technology that punts on the most difficult part of distributed stream processing and makes it the consumer's problem.
What's the most difficult part of distributed stream processing?
That is sort of danced around a bit in this article where the author is talking about dropped messages, etc. It is tempting to say "use a stream server" but ultimately stream servers make head-of-line accounting the consumer's responsibility. That's usually solved with some kind of (not distributed) lock.
Earlier quoted context omitted.
Is the title supposed to say "(How) I solved a distributed queue problem 15 years ago"? As it stands it makes it sound like you've been working on this problem for 15 years and just solved it.
Yes, that is the title, but HN deletes the word "How" from the beginning of any title.
Feels like its written by AI, has no soul.
I can promise you that I am not an AI. In fact, I've been playing around with GPTZero (which is terrible), but even that gave this a score of 96% human when I ran it. No AI was used in creating this content. Your accusation of me being soulless warrants further investigation though.
Earlier quoted context omitted.
I can promise you that I am not an AI. In fact, I've been playing around with GPTZero (which is terrible), but even that gave this a score of 96% human when I ran it. No AI was used in creating this content. Your accusation of me being soulless warrants further investigation though.
Do you recall selling it at some point in the past?
Earlier quoted context omitted.
Do you recall selling it at some point in the past?
I don't specifically recall selling it, but my life is pretty blessed, so there is certainly a chance that I made a deal with devil.