Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Hatchet – Open-source distributed task queue

github.com

Post reply on HN
Print view View original

151–160 of 195 posts

Re: Show HN: Hatchet – Open-source distributed task queue

#151

Post by hosh »

zepolen wrote: post #149
cpursley wrote: post #140

Earlier quoted context omitted.

Yep, it feels like half the show hn launches is for infrastructure tooling that already exist natively or as plug and play libraries for Elixir/Erlang. I really try to suggest people skip Node and learn a proper backend language with a solid framework with a proven architecture.

Oban looks great, how would one run a python cuda based workload on it?

You could shell out to execute with porcelain, make the python a long-running process and use ports, or port your python code to NX.

Re: Show HN: Hatchet – Open-source distributed task queue

#152

Post by hosh »

jaggederest wrote: post #90
abelanger wrote: post #21

Earlier quoted context omitted.

Thank you, appreciate the kind words! What boxes are you looking to check? Yes, I'm not a fan of the RabbitMQ dependency either - see here for the reasoning: https://news.ycombinator.com/item?id=39643940 . It would take some work to replace this with listen/notify in Postgres, less work to replace this with an in-memory component, but we can't provide the same guarantees in that case.

I come to this only as an interested observer, but my experience with listen/notify is that it outperforms rabbitmq/kafka in small to medium operations and has always pleasantly surprised me. You might find out it's a little easier than you think to slim your dependency stack down.

How do you handle things when no listeners are available to be notified?

Re: Show HN: Hatchet – Open-source distributed task queue

#153

Post by hosh »

danielovichdk wrote: post #130
necovek wrote: post #121

Earlier quoted context omitted.

There is some implicit context you are missing here. Tools like hatchet are one less dependency for projects already using Postgres: Postgres has become a de-facto database to build against. Compare that to an application built on top of Postgres and using Celery + Redis/RabbitMQ. Also, it seems like you are confusing aesthetic with ergonomics. Since forever, software developers have tried to improve on all of "aesth…

"Since forever, software developers have tried to improve on all of "aesthetics" (code/system structure appearance), "ergonomics" (how easy/fast is it to build with) and "performance" (how well it works), and the cycle has been continuous" Fast,easy,well,cheap is not a quality measure but it sure is a way to build more useless abstractions. You tell me which abstractions has made your software twice as effective.

Efficacy has more to do with the specific situation than the tools you use. Rather, it is versatility of a tool that allows someone to take advantage of the situation.

What makes abstractions more versatile has more to do with its composability and expressiveness of those compositions.

An abstraction that attempts to (apparently) reduce complexity without also being composable, is overall less versatile. Usually, something that does one thing well, is designed to also be as simple as possible. Otherwise you are increasing the overall complexity (and reducing reliability or making it fragile instead of anti-fragile) for very little gain.

Re: Show HN: Hatchet – Open-source distributed task queue

#154

Post by cebert »

The website for Hatchet and the GitHub repository make it look like a compelling distributed task queue solution. I see from the main website that this appears to have commercial aspirations, but I don’t see any pricing information available. Do you have a pricing model yet? I’d be apprehensive to consider using Hatchet in future projects without knowing how much it costs.

Re: Show HN: Hatchet – Open-source distributed task queue

#155

Post by otabdeveloper4 »

doctorpangloss wrote: post #52
kcorbitt wrote: post #17

I love your vision and am excited to see the execution! I've been looking for exactly this product (postgres-backed task queue with workers in multiple languages and decent built-in observability) for like... 3 years. Every 6 months I'll check in and see if someone has built it yet, evaluate the alternatives, and come away disappointed. One important feature request that probably would block our adoption: one reason…

Why does the RabbitMQ dependency matter? It was pretty painless for me to set up and write tests against. The operator works well and is really simple if you want to save money. I mean, isn’t Hatchett another dependency? Graphile Worker? I like all these things, but why draw the line at one thing over another over essentially aesthetics? You better start believing in dependencies if you’re a programmer.

> You better start believing in dependencies if you’re a programmer.

Yeah, faith will be your last resort when the resulting tower of babel fails in hitherto unknown to man modes.

Re: Show HN: Hatchet – Open-source distributed task queue

#156

Post by dangoodmanUT »

abelanger wrote: post #29
bluehadoop wrote: post #15

How does this compare against Temporal/Cadence/Conductor? Does hatchet also support durable execution? https://temporal.io/ https://cadenceworkflow.io/ https://conductor-oss.org/

It's very similar - I used Temporal at a previous company to run a couple million workflows per month. The gRPC networking with workers is the most similar component, I especially liked that I only had to worry about an http2 connection with mTLS instead of a different broker protocol. Temporal is a powerful system but we were getting to the point where it took a full-time engineer to build an observability layer aro…

> we were getting to the point where it took a full-time engineer to build an observability layer around Temporal

We did it in like 5 minutes by adding in otel traces? And maybe another 15 to add their grafana dashboard?

What obstacles did you experience here?

Re: Show HN: Hatchet – Open-source distributed task queue

#158

Post by HoyaSaxa »

rubenfiszel wrote: post #148

Ola, fellow YC founders. Surely you have seen Windmill since you refer to it in the comments below. It looks like Hatchet, being a lot more recent, has currently a subset of what Windmill offers, albeit with a focus solely on the task queue and without the self-hosted enterprise focus. So it looks more like a competitor to Inngest than of Windmill. We released workflows as code last week which was the primary differe…

No connection to either company, but for what it’s worth I’d never in a million years consider Windmill and this product to be direct competitors.

We’ve had a lot of pain with celery and Redis over the years and Hatchet seems to be a pretty compelling alternative. I’d want to see the codebase stabilize a bit before seriously considering it though. And frankly I don’t see a viable path to real commercialization for them so I’d only consider it if everything you needed really was MIT licensed.

Windmill is super interesting but I view it as the next evolution of something like Zapier. Having a large corpus of templates and integrations is the power of that type of product. I understand that under the hood it is a similar paradigm, but the market positioning is rightfully night and day. And I also do see a path to real commercialization of the Windmill product because of the above.

Re: Show HN: Hatchet – Open-source distributed task queue

#159

Post by sixdimensional »

ako wrote: post #70
kcorbitt wrote: post #17

I love your vision and am excited to see the execution! I've been looking for exactly this product (postgres-backed task queue with workers in multiple languages and decent built-in observability) for like... 3 years. Every 6 months I'll check in and see if someone has built it yet, evaluate the alternatives, and come away disappointed. One important feature request that probably would block our adoption: one reason…

Funny how this is vision now. I started my career 29 years ago at a company that build exactly this, but based on oracle. The agents would run on Solaris, aix, vax vms, hpux, windows nt, iris, etc. Was also used to create an automated cicd pipeline to build all binaries on all these different systems.

Because people don’t know what they don’t know, and, learning from others (along with human knowledge sharing and transfer) doesn’t seem to be what society often prioritizes in general.

Not so much talking about the original post, I think it’s awesome what they are building, and clearly they have learned by observing other things.

Re: Show HN: Hatchet – Open-source distributed task queue

#160

Post by rubenfiszel »

HoyaSaxa wrote: post #158
rubenfiszel wrote: post #148

Ola, fellow YC founders. Surely you have seen Windmill since you refer to it in the comments below. It looks like Hatchet, being a lot more recent, has currently a subset of what Windmill offers, albeit with a focus solely on the task queue and without the self-hosted enterprise focus. So it looks more like a competitor to Inngest than of Windmill. We released workflows as code last week which was the primary differe…

No connection to either company, but for what it’s worth I’d never in a million years consider Windmill and this product to be direct competitors. We’ve had a lot of pain with celery and Redis over the years and Hatchet seems to be a pretty compelling alternative. I’d want to see the codebase stabilize a bit before seriously considering it though. And frankly I don’t see a viable path to real commercialization for th…

Windmill is used by large enterprises to run critical jobs that require a predefined amount of resources and can run for months if needed, stream their logs, written in code at scale with upmost reliability, throughput and lowest overhead. The only insight from Zapier is how easy it is to develop new workflows.

I understand our positioning is not clear on our landing (and we are working on it), but my read of hatched is that what they put forward is mostly a durable execution engine for arbitrary code in python/typescript on a fleet of managed workers, which is exactly what Windmill is. We are profitable and probably wouldn't if we were MIT licensed with no enterprise features.

From reading their documentation, the implementation is extremely similar, you define workflows as code ahead of time, and then the engine make sure to have them progress reliably on your fleet of workers (one of our customer has 600 workers deployed on edge environments). There are a few minor differences, we implement the workers as generic rust binary that pull the workflows, so you never have to redeploy them to test and deploy new workflows, whereas they have developed SDK for each languages to allow you to define your own deployable workers (which is more similar to Inngest/Temporal). Also we use polling and REST instead of gRPC for communications between workers and servers.

Post reply on HN