Looks cool, but I’m still team everything-in-Postgres
This uses Postgres
And RabbitMQ :S
Launch HN: Hatchet (YC W24) – Open-source task queue, now with a cloud version
91–100 of 102 posts
Re: Launch HN: Hatchet (YC W24) – Open-source task queue, now with a cloud version
#92Hatchet looks pretty awesome. I was thinking about using it to replace my Celery worker. However, the problem is that I can only use the gRPC client to create a task (correct me if I am wrong). What I want is to be able to commit a bunch of database rows altogether with the background task itself directly. The benefit of doing so with a PostgreSQL database is that all the rows will be in the same transaction. With tr…
That's correct, you can only create tasks via the gRPC client, Hatchet can't hook into the same transaction as your inserts or updates. It seems like a very lightweight tasks table in your existing PG database which represents whether or not the task has been written to Hatchet would solve both of these cases. Once Hatchet is sent the workflow/task to execute, it's guaranteed to be enqueued/requeued. That way, you co…
I wonder if you could use LISTEN/NOTIFY so when a task name and payload are committed to a 'tasks' table, then it enqueues the job as if you had done so via gRPC?
Re: Launch HN: Hatchet (YC W24) – Open-source task queue, now with a cloud version
#93Can somebody explain why would I use it instead a simple Redis/SQS/Postgres queue implemented in 50 LOC (+ some grafana panel for monitoring) (which is pretty much mandatory even for a wrapper of this or any other service)? I'm not trying to mock it, it's a serious question. What is implied by "task queue" that makes it worth bothering to use a dedicated service?
You can use celery with postgres without issues if you want the stuff you don't get with that, like tweakable retries, tweakable amounts of prefetch and other important-at-scale things. Plus out of the box working sdk with higher level patterns for you developers. Like what if devs want to track how long something waited in the queue or a metric about retries etc, things that you'd have to roll by hand.
> You can use celery with postgres without issues
How? This issue still seems to be open after 6 years: https://github.com/celery/celery/issues/5149
Re: Launch HN: Hatchet (YC W24) – Open-source task queue, now with a cloud version
#94How is this technically different from what solid queue is doing on rails ?
https://github.com/rails/solid_queue
Just trying to understand. I do get that hatchet would be language agnostic, SDK API kind of a solution.
Re: Launch HN: Hatchet (YC W24) – Open-source task queue, now with a cloud version
#95Can somebody explain why would I use it instead a simple Redis/SQS/Postgres queue implemented in 50 LOC (+ some grafana panel for monitoring) (which is pretty much mandatory even for a wrapper of this or any other service)? I'm not trying to mock it, it's a serious question. What is implied by "task queue" that makes it worth bothering to use a dedicated service?
You're right, if all you need is a queue with a small number of workers connected at low volume, you don't need Hatchet or any other managed queue - you can get some pretty performant behavior with something like: https://github.com/abelanger5/postgres-fair-queue/blob/main/....
The point of Hatchet is to support more complex behavior - like chaining tasks together, building automation around querying and retrying failed tasks, handling a lot of the fairness and concurrency use-cases you'd otherwise need to build yourself, etc - or just getting something that works out of the box and can support those use-cases in the future.
And if you are running at low volume and trying to debug user issues, a grafana panel isn't going to get you the level of granularity or admin control you need to track down the errors in your methods (rather than just at the queue level). You'd need to integrate your task queue with Sentry and a logging system - and in our case, error tracing and logging are available in the Hatchet UI.
Re: Launch HN: Hatchet (YC W24) – Open-source task queue, now with a cloud version
#96Can somebody explain why would I use it instead a simple Redis/SQS/Postgres queue implemented in 50 LOC (+ some grafana panel for monitoring) (which is pretty much mandatory even for a wrapper of this or any other service)? I'm not trying to mock it, it's a serious question. What is implied by "task queue" that makes it worth bothering to use a dedicated service?
You're right, if all you need is a queue with a small number of workers connected at low volume, you don't need Hatchet or any other managed queue - you can get some pretty performant behavior with something like: https://github.com/abelanger5/postgres-fair-queue/blob/main/... . The point of Hatchet is to support more complex behavior - like chaining tasks together, building automation around querying and retrying fa…
That caught my attention. Retrying failed tasks isn’t easy. There are all kinds of corner cases that pop up one by one. If you have some nice way to handle the common failure modes ("text me" or "retry every ten minutes" or "retry 5 times, then give up" or "keep retrying, but with exponential backoff") then that’s something I’d love to use.
(Wiring together 40+ preemptible TPUs was a nice crucible for learning about all of these. And much like a crucible, it was as painful as it sounds. Hatchet would’ve been nice.)
Thanks for making this!
Re: Launch HN: Hatchet (YC W24) – Open-source task queue, now with a cloud version
#97Earlier quoted context omitted.
You're right, if all you need is a queue with a small number of workers connected at low volume, you don't need Hatchet or any other managed queue - you can get some pretty performant behavior with something like: https://github.com/abelanger5/postgres-fair-queue/blob/main/... . The point of Hatchet is to support more complex behavior - like chaining tasks together, building automation around querying and retrying fa…
That caught my attention. Retrying failed tasks isn’t easy. There are all kinds of corner cases that pop up one by one. If you have some nice way to handle the common failure modes ("text me" or "retry every ten minutes" or "retry 5 times, then give up" or "keep retrying, but with exponential backoff") then that’s something I’d love to use. (Wiring together 40+ preemptible TPUs was a nice crucible for learning about…
Yep, we have support for "retry 5 times, then give up" (https://docs.hatchet.run/home/features/retries/simple) and "text me" - you can use either our built-in alerting features which integrate with email and Slack, or configure your own on failure step (https://docs.hatchet.run/home/features/on-failure-step).
Configurable retry delays are currently in development.
Re: Launch HN: Hatchet (YC W24) – Open-source task queue, now with a cloud version
#98Earlier quoted context omitted.
Do you find it frustrating that what people basically want is: (1) you, for free (2) develop all the functionality of RabbitMQ as a Postgres extension with the most permissive license (3) in order to have it on RDS (4) and never hear from you again? This is a colorful exaggeration. But it’s true. It is playing out with the pgvecto-rs people too. People don’t want Postgres because it is good. They want it because it i…
So true. The advice of "commoditize your complements" is working out great for amazon. Ironically, AWS is almost a commodity itself, and the OSS community could flip the table, but we haven't figured out how to do it.
Similar to Amazon's Retail fulfillment infrastructure, the AWS supply chain infrastructure is definitely not a commodity.
Re: Launch HN: Hatchet (YC W24) – Open-source task queue, now with a cloud version
#99I was going to reach for dagster to run some DBT and data load scripts. No orchestration as of yet-why would I use hatchet instead of dagster?
Re: Launch HN: Hatchet (YC W24) – Open-source task queue, now with a cloud version
#100Earlier quoted context omitted.
After multiple years fighting with Celery, we moved to Prefect last year and have been mostly happy with it. The only sticking point for me has been “tasks can’t start tasks, will have to be sub-flows” part. Did you ever try out Prefect and can share anything from the experience?
I don't have any experience with prefect, but I have to say one of my favorite things about SAQ (Simple Async Queue) was a task was a task was a task. You could enqueue them from anywhere, nest them, repeat them, skip them, whichever. With hatchet theres been a little bit of a dance trying to get workflows and runs to play nicely, but all in all I was able to get everything I needed working without much trouble. You…
I see. Thanks for sharing.