Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Hatchet – Open-source distributed task queue

github.com

Post reply on HN
Print view View original

101–110 of 195 posts

Re: Show HN: Hatchet – Open-source distributed task queue

#101

Post by renegade-otter »

bevekspldnw wrote: post #75
kcorbitt wrote: post #17

I love your vision and am excited to see the execution! I've been looking for exactly this product (postgres-backed task queue with workers in multiple languages and decent built-in observability) for like... 3 years. Every 6 months I'll check in and see if someone has built it yet, evaluate the alternatives, and come away disappointed. One important feature request that probably would block our adoption: one reason…

You can do a fair amount of this with Postgres using locks out of the box. It’s not super intuitive but I’ve been using just Postgres and locks in production for many years for large task distribution across independent nodes.

I wrote about one simple implementation:

https://renegadeotter.com/2023/11/30/job-queues-with-postrgr...

Re: Show HN: Hatchet – Open-source distributed task queue

#102

Post by Nukesor »

Hey @abelanger,

I got a few feature request for Pueue that were out of the scope as they didn't fit Pueue's vision, but seem to fit hatchet quite well (e.g. complex scheduling functionality and multi-agent support) :)

One thing I'm missing from your website however, is an actual view from how the interface looks like, what does the actual user interface look like.

Having the possibility to schedule stuff in a smart way is nice and all, but how do you *overlook* it? It's important to get a good overview of how your tasks perform.

Once I'm convinced that this is actually a useful piece of software, I would like to reference you in the Readme of Pueue as a alternative for users that need more powerful scheduling features (or multi-client support) :) Would that be ok for you?

Re: Show HN: Hatchet – Open-source distributed task queue

#103

Post by zwaps »

abelanger wrote: post #55
zwaps wrote: post #46

You say this is for generative AI. How do you distribute inference across workers? Can one use just any protocol and how does this work together with the queue and fault tolerance? Could not find any specifics on generative AI in your docs. Thanks

This isn't built specifically for generative AI, but generative AI apps typically have architectural issues that are solved by a good queueing system and worker pool. This is particularly true once you start integrating smaller, self-hosted LLMs or other types of models into your pipeline. > How do you distribute inference across workers? In Hatchet, "run inference" would be a task. By default, tasks get randomly ass…

Got it, so the underlying infrastructure (the inference nodes, if you wish) would be something to be solved outside of Hatched, but it would then allow to schedule inference tasks per user with limits.

Re: Show HN: Hatchet – Open-source distributed task queue

#107

Post by bevekspldnw »

renegade-otter wrote: post #101
bevekspldnw wrote: post #75

Earlier quoted context omitted.

You can do a fair amount of this with Postgres using locks out of the box. It’s not super intuitive but I’ve been using just Postgres and locks in production for many years for large task distribution across independent nodes.

I wrote about one simple implementation: https://renegadeotter.com/2023/11/30/job-queues-with-postrgr...

Looks very similar to my solution. :-)

Re: Show HN: Hatchet – Open-source distributed task queue

#108

Post by abelanger »

Nukesor wrote: post #102

Hey @abelanger, I got a few feature request for Pueue that were out of the scope as they didn't fit Pueue's vision, but seem to fit hatchet quite well (e.g. complex scheduling functionality and multi-agent support) :) One thing I'm missing from your website however, is an actual view from how the interface looks like, what does the actual user interface look like. Having the possibility to schedule stuff in a smart w…

Pueue looks cool, it's not an alternative to Hatchet though - looks like it's meant to be run in the terminal or by a user? We're very much meant to run in an application runtime.

Like I mentioned here [1], we'll expand our comparison section over time. If Pueue's an alternative people are asking about, we'll definitely put it in there.

> Having the possibility to schedule stuff in a smart way is nice and all, but how do you overlook it? It's important to get a good overview of how your tasks perform.

I'm not sure what you mean by this. Perhaps you're referring to this - https://news.ycombinator.com/item?id=39647154 - in which case I'd say: most software is far from perfect. Our scheduling works but has limitations and is being refactored before we advertise it and build it into our other SDKs.

[1] https://news.ycombinator.com/item?id=39643631

Re: Show HN: Hatchet – Open-source distributed task queue

#110

Post by Aeolun »

MuffinFlavored wrote: post #36
leetrout wrote: post #35

Just pointing out even though this is a "Show HN" they are, indeed, backed by YC. Is this going to follow the "open core" pattern or will there be a different path to revenue?

> path to revenue There have to be at least 10 different ways between different cloud providers to run a distributed task queue. Amazon, Azure, GCP Self-hosting RabbitMQ, etc. I'm curious how they are able to convince investors that there is a sizable portion of market they think doesn't already have this solved (or already has it solved and is willing to migrate)

> I'm curious how they are able to convince investors that there is a sizable portion of market they think doesn't already have this solved

Is there any task queue you are completely happy with?

I use Redis, but it’s only half of the solution.

Post reply on HN