Show HN: Hatchet – Open-source distributed task queue • Hacker News Board

Re: Show HN: Hatchet – Open-source distributed task queue

#21

Post by abelanger » Fri, Mar 08, 2024, 6:23 PM UTC

kcorbitt wrote: post #17

I love your vision and am excited to see the execution! I've been looking for exactly this product (postgres-backed task queue with workers in multiple languages and decent built-in observability) for like... 3 years. Every 6 months I'll check in and see if someone has built it yet, evaluate the alternatives, and come away disappointed. One important feature request that probably would block our adoption: one reason…

Thank you, appreciate the kind words! What boxes are you looking to check?

Yes, I'm not a fan of the RabbitMQ dependency either - see here for the reasoning: https://news.ycombinator.com/item?id=39643940.

It would take some work to replace this with listen/notify in Postgres, less work to replace this with an in-memory component, but we can't provide the same guarantees in that case.

Re: Show HN: Hatchet – Open-source distributed task queue

#22

Post by toddmorey » Fri, Mar 08, 2024, 6:27 PM UTC

I need task queues where the client (web browser) can listen to the progress of the task through completion.

I love the simplicity & approachability of Deno queues for example, but I’d need to roll my own way to subscribe to task status from the client.

Wondering if perhaps the Postgres underpinnings here would make that possible.

EDIT: seems so! https://docs.hatchet.run/home/features/streaming

Re: Show HN: Hatchet – Open-source distributed task queue

#23

Post by rad_gruchalski » Fri, Mar 08, 2024, 6:32 PM UTC

toddmorey wrote: post #22

I need task queues where the client (web browser) can listen to the progress of the task through completion. I love the simplicity & approachability of Deno queues for example, but I’d need to roll my own way to subscribe to task status from the client. Wondering if perhaps the Postgres underpinnings here would make that possible. EDIT: seems so! https://docs.hatchet.run/home/features/streaming

If you need to listen for the progress only, try server-sent events, maybe?: https://en.wikipedia.org/wiki/Server-sent_events

It's dead simple: an existence of the URI means the topic/channel/whathaveu exists, to access it one needs to know the URI, data streamed but no access to old data, multiple consumers no problem.

Re: Show HN: Hatchet – Open-source distributed task queue

#24

Post by abelanger » Fri, Mar 08, 2024, 6:34 PM UTC

pyrossh wrote: post #12

How is this different from pg-boss[1]? Other than the distributed part it also seems to use skip locked. [1] https://github.com/timgit/pg-boss

I haven't used pg-boss, and feature-wise it looks very similar and is an impressive project.

The core difference is that pg-boss is a library while Hatchet is a separate service which runs independently of your workers. This service also provides a UI and API for interacting with Hatchet - I don't think pg-boss has those things, so you'd probably have to build out observability yourself.

This doesn't make a huge difference when you're at 1 worker, but having each worker poll your database can lead to DB issues if you're not careful - I've seen some pretty low-throughput setups for very long-running jobs using a database with 60 CPUs because of polling workers. Hatchet distributes in two layers - the "engine" and the "worker" layer. Each engine polls the database and fans out to the workers over a long-lived gRPC connection. This reduces pressure on the DB and lets us manage which workers to assign tasks to based on things like max concurrent runs on each worker or worker health.

Re: Show HN: Hatchet – Open-source distributed task queue

#25

Post by abelanger » Fri, Mar 08, 2024, 6:39 PM UTC

toddmorey wrote: post #22

I need task queues where the client (web browser) can listen to the progress of the task through completion. I love the simplicity & approachability of Deno queues for example, but I’d need to roll my own way to subscribe to task status from the client. Wondering if perhaps the Postgres underpinnings here would make that possible. EDIT: seems so! https://docs.hatchet.run/home/features/streaming

Yep, exactly - Gabe has also been thinking about providing per-user signed URLs to task executions so clients can subscribe more easily without a long-lived token. So basically, you would start the workflow from your API, and pass back the signed URL to the client, where we would then provide a React hook to get task updates automatically. We need this ourselves once we open our cloud instance up to self-serve, since we want to provision separate queues per user, with a Hatchet workflow of course.

Re: Show HN: Hatchet – Open-source distributed task queue

#26

Post by toddmorey » Fri, Mar 08, 2024, 6:42 PM UTC

abelanger wrote: post #25

toddmorey wrote: post #22

I need task queues where the client (web browser) can listen to the progress of the task through completion. I love the simplicity & approachability of Deno queues for example, but I’d need to roll my own way to subscribe to task status from the client. Wondering if perhaps the Postgres underpinnings here would make that possible. EDIT: seems so! https://docs.hatchet.run/home/features/streaming

Yep, exactly - Gabe has also been thinking about providing per-user signed URLs to task executions so clients can subscribe more easily without a long-lived token. So basically, you would start the workflow from your API, and pass back the signed URL to the client, where we would then provide a React hook to get task updates automatically. We need this ourselves once we open our cloud instance up to self-serve, since…

Awesome to hear!

Re: Show HN: Hatchet – Open-source distributed task queue

#27

Post by spenczar5 » Fri, Mar 08, 2024, 6:49 PM UTC

abelanger wrote: post #14

topicseed wrote: post #2

What specific strategies does Hatchet employ to guarantee fault tolerance and enable durable execution? How does it handle partial failures in multi-step workflows?

Each task in Hatchet is backed by a workflow [1]. Workflows are predefined steps which are persisted in PostgreSQL. If a worker dies or crashes midway through (stops heartbeating to the engine), we reassign tasks (assuming they have retries left). We also track timeouts in the database, which means if we miss a timeout, we simply retry after some amount of time. Like I mentioned in the post, we avoid some classes of…

What happens if a worker goes silent for longer than the heartbeat duration, then a new worker is spawned, then the original worker “comes back to life”? For example, because there was a network partition, or because the first worker’s host machine was sleeping, or even just that the first worker process was CPU starved?

Re: Show HN: Hatchet – Open-source distributed task queue

#28

Post by jerrygenser » Fri, Mar 08, 2024, 6:50 PM UTC

Something I really like about some pub/sub systems is Push subscriptions. For example in GCP pub/sub you can have a "subscriber" that is not pulling events off the queue but instead is an http endpoint where events are pushed to.

The nice thing about this is that you can use a runtime like cloud run or lambda and allow that runtime to scale based on http requests and also scale to zero.

Setting up autoscaling for workers can be a little bit more finicky, e.g. in kubernetes you might set up KEDA autoscaling based on some queue depth metrics but these might need to be exported from rabbit.

I suppose you could have a setup where your daemon worker is making http requests and in that sense "push" to the place where jobs are actually running but this adds another level of complexity.

Is there any plan to support a push model where you can push jobs into http and some daemons that are holding the http connections opened?

Re: Show HN: Hatchet – Open-source distributed task queue

#29

Post by abelanger » Fri, Mar 08, 2024, 6:50 PM UTC

bluehadoop wrote: post #15

How does this compare against Temporal/Cadence/Conductor? Does hatchet also support durable execution? https://temporal.io/ https://cadenceworkflow.io/ https://conductor-oss.org/

It's very similar - I used Temporal at a previous company to run a couple million workflows per month. The gRPC networking with workers is the most similar component, I especially liked that I only had to worry about an http2 connection with mTLS instead of a different broker protocol.

Temporal is a powerful system but we were getting to the point where it took a full-time engineer to build an observability layer around Temporal. Integrating workflows in an intuitive way with OpenTelemetry and logging was surprisingly non-arbitrary. We wanted to build more of a Vercel-like experience for managing workflows.

We have a section on the docs page for durable execution [1], also see the comment on HN [2]. Like I mention in that comment, we still have a long way to go before users can write a full workflow in code in the same style as a Temporal workflow, users either define the execution path ahead of time or invoke a child workflow from an existing workflow. This is also something that requires customization for each SDK - like Temporal's custom asyncio event loop in their Python SDK [3]. We don't want to roll this out until we can be sure about compatibility with the way most people write their functions.

[1] https://docs.hatchet.run/home/features/durable-execution

[2] https://news.ycombinator.com/item?id=39643881

[3] https://github.com/temporalio/sdk-python

Re: Show HN: Hatchet – Open-source distributed task queue

#30

Post by dalberto » Fri, Mar 08, 2024, 7:01 PM UTC

I'm curious if this supports coroutines at tasks in Python. It's especially useful for genAI, and legacy queues (namely Celery) are lacking in this regard.

It would help to see a mapping of Celery to Hatchet as examples. The current examples require you to understand (and buy into) Hatchet's model, but that's hard to do without understanding how it compares to existing solutions.