Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Hatchet – Open-source distributed task queue

github.com

Post reply on HN
Print view View original

141–150 of 195 posts

Re: Show HN: Hatchet – Open-source distributed task queue

#141

Post by jerrygenser »

tonyhb wrote: post #89
jerrygenser wrote: post #28

Something I really like about some pub/sub systems is Push subscriptions. For example in GCP pub/sub you can have a "subscriber" that is not pulling events off the queue but instead is an http endpoint where events are pushed to. The nice thing about this is that you can use a runtime like cloud run or lambda and allow that runtime to scale based on http requests and also scale to zero. Setting up autoscaling for wor…

You might want to look at https://www.inngest.com for that. Disclaimer: I'm a cofounder. We released event-driven step functions about 20 months ago.

Looks cool but looks like it's only typescript. If there is a json payload, couldn't any web server handle it?

Re: Show HN: Hatchet – Open-source distributed task queue

#142

Post by adeptima »

Exciting time for distributed, transactional task queue projects built on the top of PostgreSQL!

Here's the most heavily upvoted in the past 12 months

Hatchet https://news.ycombinator.com/item?id=39643136

Inngest https://news.ycombinator.com/item?id=36403014

Windmill https://news.ycombinator.com/item?id=35920082

HN comments on Temporal.io https://github.com/temporalio https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

Internally we rant about the complexity of the above projects vs using transactional job queues libs like:

river https://news.ycombinator.com/item?id=38349716

neoq: [https://github.com/acaloiaro/neoq](https://github.com/acaloi...

gue: [https://github.com/vgarvardt/gue](https://github.com/vgarvar...

Deep inside can't wait to see some like ThePrimeTimeagen to review it ;) https://www.youtube.com/@ThePrimeTimeagen

Re: Show HN: Hatchet – Open-source distributed task queue

#143

Post by lysecret »

jerrygenser wrote: post #28

Something I really like about some pub/sub systems is Push subscriptions. For example in GCP pub/sub you can have a "subscriber" that is not pulling events off the queue but instead is an http endpoint where events are pushed to. The nice thing about this is that you can use a runtime like cloud run or lambda and allow that runtime to scale based on http requests and also scale to zero. Setting up autoscaling for wor…

Yep we are using cloud tasks and pub sub a lot. Another big benefit is that the GCP infra is literally “pushing” your messages even if your infra goes down.

Re: Show HN: Hatchet – Open-source distributed task queue

#145

Post by bruth »

abelanger wrote: post #119
Kinrany wrote: post #38

With NATS in the stack, what's the advantage over using NATS directly?

I'm assuming specifically you mean Nex functions? Otherwise NATS gives you connectivity and a message queue - it doesn't (or didn't) have the concept of task executions or workflows. With regards to Nex -- it isn't fully stable and only supports Javascript/Webassembly. It's also extremely new, so I'd be curious to see how things stabilize in the coming year.

(Disclaimer: I am a NATS maintainer and work for Synadia)

The parent comment may have been referring to the fact that NATS has support for durable (and replicated) work queue streams, so those could be used directly for queuing tasks and having a set of workers dequeuing concurrently. And this is regardless if you would want to use Nex or not. Nex is indeed fairly new, but the team on is iterating on it quickly and we are dog-fooding it internally to keep stabilizing it.

The other benefits of NATS is the built-in multi-tenancy which would allow for distinct applications/teams/contexts to have an isolated set of streams and messaging. It acts as a secure namespace.

NATS supports clustering within a region or across regions. For example, Synadia hosts a supercluster in many different regions across the globe and across the three major cloud providers. As it applies to distributed work queues, you can place work queue streams in a cluster within a region/provider closest to the users/apps enqueuing the work, and then deploy workers in the same region for optimizing latency of dequeuing and processing.

Could be worth a deeper look on how much you could leverage for this use case.

Re: Show HN: Hatchet – Open-source distributed task queue

#146

Post by rubenfiszel »

kcorbitt wrote: post #17

I love your vision and am excited to see the execution! I've been looking for exactly this product (postgres-backed task queue with workers in multiple languages and decent built-in observability) for like... 3 years. Every 6 months I'll check in and see if someone has built it yet, evaluate the alternatives, and come away disappointed. One important feature request that probably would block our adoption: one reason…

Windmill is is built exactly like that, what box is left unchecked for it if you had time to review it?

Re: Show HN: Hatchet – Open-source distributed task queue

#147

Post by [deleted] »

adeptima wrote: post #142

Exciting time for distributed, transactional task queue projects built on the top of PostgreSQL! Here's the most heavily upvoted in the past 12 months Hatchet https://news.ycombinator.com/item?id=39643136 Inngest https://news.ycombinator.com/item?id=36403014 Windmill https://news.ycombinator.com/item?id=35920082 HN comments on Temporal.io https://github.com/temporalio https://hn.algolia.com/?dateRange=all&page=0&pref…

[deleted]

Re: Show HN: Hatchet – Open-source distributed task queue

#148

Post by rubenfiszel »

Ola, fellow YC founders. Surely you have seen Windmill since you refer to it in the comments below. It looks like Hatchet, being a lot more recent, has currently a subset of what Windmill offers, albeit with a focus solely on the task queue and without the self-hosted enterprise focus. So it looks more like a competitor to Inngest than of Windmill. We released workflows as code last week which was the primary differentiator with other workflow engines and us so far: https://www.windmill.dev/docs/core_concepts/workflows_as_cod...

The license is more permissive than ours MIT vs AGPLv3, and you're using Go vs Rust for us, but other than that the architecture looks extremely similar, also based mostly on Postgres with the same insights than us: it's sufficient. I'm curious where do you see the main differentiator long-term.

Re: Show HN: Hatchet – Open-source distributed task queue

#149

Post by zepolen »

cpursley wrote: post #140
throwawaymaths wrote: post #129

Earlier quoted context omitted.

Also basically has existed as an open source (pro version has web dashboard and complex task zoo) drop-in library (no sidecar dependencies outside of postgres) in Elixir for years called Oban.

Yep, it feels like half the show hn launches is for infrastructure tooling that already exist natively or as plug and play libraries for Elixir/Erlang. I really try to suggest people skip Node and learn a proper backend language with a solid framework with a proven architecture.

Oban looks great, how would one run a python cuda based workload on it?

Re: Show HN: Hatchet – Open-source distributed task queue

#150

Post by leetrout »

wodenokoto wrote: post #132
leetrout wrote: post #35

Just pointing out even though this is a "Show HN" they are, indeed, backed by YC. Is this going to follow the "open core" pattern or will there be a different path to revenue?

Wasn’t the first Dropbox introduction also a show HN? I don’t think this is out of place

I am not saying it is out of place but I feel for such a long winded explanation of what they are doing a missing "YC W24" was surprising.
Post reply on HN