Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Hatchet – Open-source distributed task queue

github.com

Post reply on HN
Print view View original

171–180 of 195 posts

Re: Show HN: Hatchet – Open-source distributed task queue

#171

Post by stephen »

Wow, looks great! We currently happily use graphile-worker, and have two questions:

> full transactional enqueueing

Do you mean transactional within the same transaction as the application's own state?

My guess is no (from looking at the docs, where enqueuing in the SDK looks a lot like a wire call and not issuing a SQL command over our application's existing connection pool), and that you mean transactionality between steps within the Hatchet jobs...

I get that, but fwiw transactionality of "perform business logic against entities + job enqueue" (both for queuing the job itself, as well as work performed by workers) is the primary reason we're using a PG-based job queue, as then we avoid transactional outboxes for each queue/work step.

So, dunno, loosing that would be a big deal/kinda defeat the purpose (for us) of a PG-based queue.

2nd question, not to be a downer, but I'm just genuinely curious as a wanna-be dev infra/tooling engineer, but a) why take funding to build this (it seems bootstrappable? maybe that's naive), and b) why would YC keeping putting money into these "look really neat but ...surely?... will never be the 100x returns/billion dollar companies" dev infra startups? Or maybe I'm over-estimating the size of the return/exit necessary to make it worth their while.

Re: Show HN: Hatchet – Open-source distributed task queue

#172

Post by abelanger »

dangoodmanUT wrote: post #156
abelanger wrote: post #29

Earlier quoted context omitted.

It's very similar - I used Temporal at a previous company to run a couple million workflows per month. The gRPC networking with workers is the most similar component, I especially liked that I only had to worry about an http2 connection with mTLS instead of a different broker protocol. Temporal is a powerful system but we were getting to the point where it took a full-time engineer to build an observability layer aro…

> we were getting to the point where it took a full-time engineer to build an observability layer around Temporal We did it in like 5 minutes by adding in otel traces? And maybe another 15 to add their grafana dashboard? What obstacles did you experience here?

Well, for one - most otel services (like Honeycomb) are designed around aggregate views, and engineers found it difficult to track down the failure of specific workflows. We were already using Sentry, had started adding prom + grafana into our stack, and were already using mezmo for logging. So to debug a workflow, we'd see an alert come in through Sentry, grab the workflow ID and activity ID, perform a search in the Temporal console, track down the failed activity (of which there could be between 1-100 activities), and associate that with our logs in mezmo (involving a new query syntax). This is a lot of raw data that takes time to parse and figure out what's going wrong. And then we wanted to build out a view of worker health, which involves a new set of dashboards and alerts that are different from our error alerting in Sentry.

Yes, this sounded broken to us too - we were aware of the promise of consolidation with an opentelemetry and a Grafana stack, but we couldn't make this transition happen cleanly, and when you're already relying on certain tools for your API it makes the transition more difficult. There's also upskilling involved in getting engineers on the team to adjust to otel when they're used to more intuitive tools like sentry and mezmo.

A good set of default metrics, better search, and views for worker performance and pools - that would have gone a long way. The extent of Temporal UI features are basic recent workflows, an expanded workflow view with stack traces for thrown errors, a schedules page, and a settings page.

Re: Show HN: Hatchet – Open-source distributed task queue

#174

Post by echelon »

abelanger wrote: post #64
leetrout wrote: post #35

Just pointing out even though this is a "Show HN" they are, indeed, backed by YC. Is this going to follow the "open core" pattern or will there be a different path to revenue?

Yep, we're backed by YC in the W24 batch - this is evident on our landing page [1]. We're both second time CTOs and we've been on both sides of this, as consumers of and creators of OSS. I was previously a co-founder and CTO of Porter [2], which had an open-core model. There are two risks that most companies think about in the open core model: 1. Big companies using your platform without contributing back in some way…

I got flagged, but I want to reiterate that you need legal means of stopping AWS from simply lifting your product wholesale. Just look at all the other companies they've turned into their own thankless premium offerings.

Put in a DAU/MAU/volume/revenue clause that pertains specifically only to hyperscalers and resellers. Don't listen to the naysayers telling you not to do it. This isn't their company or their future. They don't care if you lose your business or that you put in all of that work just for a tech giant to absorb it for free and turn it against you.

Just do it. Do it now and you won't get (astroturfed?) flack for that decision later by people who don't even have skin in the game. It's not a big deal. I would buy open core products with these protections -- it's not me you're protecting yourselves against, and I'm nowhere in the blast radius. You're trying not to die in the miasma of monolithic cloud vendors.

Re: Show HN: Hatchet – Open-source distributed task queue

#175

Post by abelanger »

stephen wrote: post #171

Wow, looks great! We currently happily use graphile-worker, and have two questions: > full transactional enqueueing Do you mean transactional within the same transaction as the application's own state? My guess is no (from looking at the docs, where enqueuing in the SDK looks a lot like a wire call and not issuing a SQL command over our application's existing connection pool), and that you mean transactionality betwe…

Yeah these are great questions.

> Do you mean transactional within the same transaction as the application's own state? My guess is no (from looking at the docs, where enqueuing in the SDK looks a lot like a wire call and not issuing a SQL command over our application's existing connection pool), and that you mean transactionality between steps within the Hatchet jobs...

Yeah, it's the latter, though we'd actually like to support the former in the longer term. There's no technical reason we can't write the workflow/task and read from the same table that you're enqueueing with in the same transaction as your application. That's the really exciting thing about the RiverQueue implementation, though it also illustrates how difficult it is to support every PG driver in an elegant way.

Transactional enqueueing is important for a whole bunch of other reasons though - like assigning workers, maintaining dependencies between tasks, implementing timeouts.

> why take funding to build this (it seems bootstrappable? maybe that's naive)

The thesis is that we can help some users offload their tasks infra with a hosted version, and hosted infra is hard to bootstrap.

> why would YC keeping putting money into these "look really neat but ...surely?... will never be the 100x returns/billion dollar companies" dev infra startups?

I think Cloudflare is an interesting example here. You could probably make similar arguments against a spam protection proxy, which was the initial service. But a lot of the core infrastructure needed for that service branches into a lot of other products, like a CDN or caching layer, or a more compelling, full-featured product like a WAF. I can't speak for YC or the other dev infra startups, but I imagine that's part of the thesis.

Re: Show HN: Hatchet – Open-source distributed task queue

#176

Post by abelanger »

wereHamster wrote: post #144

Does it (or will it, ie. is it planned) support delayed execution? eg. I have a task that I want to run at a certain time in the future?

It is planned - see here for more details: https://news.ycombinator.com/item?id=39646300.

We still need to do some work on this feature though, we'll make sure to document it when it's well-supported.

Re: Show HN: Hatchet – Open-source distributed task queue

#177

Post by stephen »

abelanger wrote: post #175
stephen wrote: post #171

Wow, looks great! We currently happily use graphile-worker, and have two questions: > full transactional enqueueing Do you mean transactional within the same transaction as the application's own state? My guess is no (from looking at the docs, where enqueuing in the SDK looks a lot like a wire call and not issuing a SQL command over our application's existing connection pool), and that you mean transactionality betwe…

Yeah these are great questions. > Do you mean transactional within the same transaction as the application's own state? My guess is no (from looking at the docs, where enqueuing in the SDK looks a lot like a wire call and not issuing a SQL command over our application's existing connection pool), and that you mean transactionality between steps within the Hatchet jobs... Yeah, it's the latter, though we'd actually li…

Nice! Thanks for the reply!

> hosted infra is hard to bootstrap.

Ah yeah, that definitely makes sense...

> a lot of the core infrastructure needed for that service branches > into a lot of other products

Ah, I think I see what you mean--the goal isn't to be "just a job queue" in 2-5 years, it's to grow into a wider platform/ecosystem/etc.

Ngl I go back/forth between rooting for dev-founded VC companies like yourself, or Benjie, the guy behind graphile-worker, who is tip-toeing into being commercially-supported.

Like I want both to win (paydays all around! :-D), but the VC money just gives such a huge edge, of establishing a very high bar of polish / UX / docs / devrel / marketing, basically using loss-leading VC money for a bet that may/may not work out, that it's very hard for the independents to compete. I have honestly been hoping post-ZIRP would swing some advantage back to the Benjie's of the world, but looks like no/not yet.

...I say all of above ^ while myself working for a VC-backed prop-tech company...so, kinda the pot calling the kettle black. :-D

Good luck! The fairness/priority queues of Hatchet definitely seem novel, at least from what I'm used to, so will keep it bookmarked/in the tool chest.

Re: Show HN: Hatchet – Open-source distributed task queue

#178

Post by jaggederest »

abelanger wrote: post #169
hosh wrote: post #152

Earlier quoted context omitted.

How do you handle things when no listeners are available to be notified?

Presumably there'd be a messages table that you listen/notify on, and you'd replay messages that weren't consumed when a listener rejoins. But yeah, this is the overhead I was referencing.

Yep, but practically speaking, you need those records anyway even if you're using another queue to actually distribute the jobs. At least every system I've ever built of a reasonable size has a job audit table anyway. Plus it's an "Enterprise Feature™" so you can differentiate on it if you like that kind of feature-based pricing

Re: Show HN: Hatchet – Open-source distributed task queue

#179

Post by mfrye0 »

abelanger wrote: post #114
mfrye0 wrote: post #69

I've been looking for this exact thing for awhile now. I'm just starting to dig into the docs and examples, and I have a question on workflows. I have an existing pipeline that runs tasks across two K8 clusters and share a DB. Is it possible to define steps in a workflow where the step run logic is setup to run elsewhere? Essentially not having an inline run function defined, and another worker process listening for…

This depends on the SDK - both Typescript and Golang support a `registerAction` method on the worker which basically let you register a single step to only run on that worker. You would then call `putWorkflow` programmatically before starting the worker. Steps are distributed by default so they run on the workers which have registered them. Happy to provide a more concrete example for the language you're using.

Perfect. Yeah, we're using both, but mainly TS. We'll test that out.

Re: Show HN: Hatchet – Open-source distributed task queue

#180

Post by Kinrany »

abelanger wrote: post #119
Kinrany wrote: post #38

With NATS in the stack, what's the advantage over using NATS directly?

I'm assuming specifically you mean Nex functions? Otherwise NATS gives you connectivity and a message queue - it doesn't (or didn't) have the concept of task executions or workflows. With regards to Nex -- it isn't fully stable and only supports Javascript/Webassembly. It's also extremely new, so I'd be curious to see how things stabilize in the coming year.

I wasn't thinking of Nex, I didn't realize Hatchet includes compute and doesn't just store tasks.

Still, it seems like NATS + any lambda implementation + a dumb service that wakes lambdas when they need to process something, would be simple to set up and in combination do the same thing.

Post reply on HN