Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Hatchet (YC W24) – Open-source task queue, now with a cloud version

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 102 posts

Re: Launch HN: Hatchet (YC W24) – Open-source task queue, now with a cloud version

#21

Post by abelanger »

nickzelei wrote: post #11

Interesting and congrats on the launch! I am definitely a fan of all things postgres and it's great to see another solution that uses it. My main thing is the RabbitMQ dependency (that seems to be a topic of interest in this thread). Getting rid of that and just depending on PG seems like the main path forward that would increase adoption. Right now I'd be considering something like this over using a tool like Rabbit…

> My main thing is the RabbitMQ dependency (that seems to be a topic of interest in this thread). Getting rid of that and just depending on PG seems like the main path forward that would increase adoption.

Yep, we agree - this is more a matter of bandwidth as well as figuring out the final definition of the pub/sub interface. While we wouldn't prefer to maintain two message queue implementations, we likely won't drop the RabbitMQ implementation entirely, even if we offer Postgres as an alternative. So if we do need to support two implementations, we'd prefer to build out a core set of features that we're happy with first. That said, the message queue API is definitely stabilizing (https://github.com/hatchet-dev/hatchet/blob/31cf5be248ff9ed7...), so I hope we can pick this up in the coming months.

> You also compare yourself against Celery and BullMQ, but there is also talk in the readme around durable execution. That to me puts you in the realm of Temporal. How would you say you compare/compete with Temporal? Are you looking to compete with them?

Yes, our child workflows feature is an alternative to Temporal which lets you execute Temporal-like workflows. These are durable from the perspective of the parent step which executes them, as any events generated by the child workflows get replayed if the parent step re-executes. Non-parent steps are the equivalent of a Temporal activity, while parent steps are the equivalent of a Temporal workflow.

Our longer-term goal is to build a better developer experience than Temporal, centered around observability and worker management. On the observability side, we're investing heavily in our dashboard, eventing, alerting and logging features. On the worker management side, we'd love to integrate more natively with worker runtime environments to handle use-cases like autoscaling.

Re: Launch HN: Hatchet (YC W24) – Open-source task queue, now with a cloud version

#22

Post by giovannibonetti »

fizx wrote: post #15
doctorpangloss wrote: post #8

Earlier quoted context omitted.

Do you find it frustrating that what people basically want is: (1) you, for free (2) develop all the functionality of RabbitMQ as a Postgres extension with the most permissive license (3) in order to have it on RDS (4) and never hear from you again? This is a colorful exaggeration. But it’s true. It is playing out with the pgvecto-rs people too. People don’t want Postgres because it is good. They want it because it i…

So true. The advice of "commoditize your complements" is working out great for amazon. Ironically, AWS is almost a commodity itself, and the OSS community could flip the table, but we haven't figured out how to do it.

AWS is a commodity, albeit an expensive one. After all, it has competitors like GCP, which some people like me actually prefer.

Re: Launch HN: Hatchet (YC W24) – Open-source task queue, now with a cloud version

#24

Post by fangpenlin »

Hatchet looks pretty awesome. I was thinking about using it to replace my Celery worker. However, the problem is that I can only use the gRPC client to create a task (correct me if I am wrong). What I want is to be able to commit a bunch of database rows altogether with the background task itself directly. The benefit of doing so with a PostgreSQL database is that all the rows will be in the same transaction. With traditional background worker solutions, you will run into two problems:

1. Commit changes in the db first: if you fail to enqueue the task, there will be data rows hanging in the db but no task to process them

2. Push the task first: the task may kick start too early, and the DB transaction is not committed yet, it cannot find the rows still in transaction. You will need to retry failure

We also looked at Celery and hope it can provide a similar offer, but the issue seems open for years:

https://github.com/celery/celery/issues/5149

With the needs, I build a simple Python library on top of SQLAlchemy:

https://github.com/LaunchPlatform/bq

It would be super cool if Hatchet also supports native SQL inserts with ORM frameworks. Without the ability to commit tasks with all other data rows, I think it's missing out a bit of the benefit of using a database as the worker queue backend.

Re: Launch HN: Hatchet (YC W24) – Open-source task queue, now with a cloud version

#25

Post by abelanger »

doctorpangloss wrote: post #8
abelanger wrote: post #6

Earlier quoted context omitted.

We're using RabbitMQ for pub/sub between different components of our engine. The actual task queue is entirely backed by Postgres, but things like streaming events between different workers are done through RabbitMQ at the moment, as well as sending a message from one component to another when you distribute the engine components. I've written a little more about this here: https://news.ycombinator.com/item?id=396439…

Do you find it frustrating that what people basically want is: (1) you, for free (2) develop all the functionality of RabbitMQ as a Postgres extension with the most permissive license (3) in order to have it on RDS (4) and never hear from you again? This is a colorful exaggeration. But it’s true. It is playing out with the pgvecto-rs people too. People don’t want Postgres because it is good. They want it because it i…

While I understand the sentiment, we see it very differently. We're interested in creating the best product possible, and being open source helps with that. The users who are self-hosting in our Discord give extremely high quality feedback and post feature ideas and discussions which shape the direction of the product. There's plenty of room for Hatchet the OSS repo and Hatchet the cloud version to coexist.

> develop all the functionality of RabbitMQ as a Postgres extension with the most permissive license

That's fair - we're not going to develop all the functionality of RabbitMQ on Postgres (if we were, we probably would have started with a amqp-compatible broker). We're building the orchestration layer that sits on top of the underlying message queue and database to manage the lifecycle of a remotely-invoked function.

Re: Launch HN: Hatchet (YC W24) – Open-source task queue, now with a cloud version

#26

Post by pm90 »

cyral wrote: post #23

How does this compare to Temporal or Inngest? I've been investigating them and the durable execution pattern recently and would like to implement one soon.

Temporal is kinda difficult to self host. Plus you have to buy into their specific paradigm/terminology for running tasks. This tool seems a lot more generic.

Re: Launch HN: Hatchet (YC W24) – Open-source task queue, now with a cloud version

#28

Post by dalberto »

abelanger wrote: post #6
dalberto wrote: post #3

I'm super interested in a Postgres-only task queue, but I'm still unclear from your post whether the only broker dependency is PostgreSQL. You mention working towards getting rid of the RabbitMQ dependency but the existence of RabbitMQ in your stack is dissonant with the statement 'a conviction that PostgreSQL is the right choice for a task queue'. In my mind, if you are using Postgres as a queue, I'm not sure why yo…

We're using RabbitMQ for pub/sub between different components of our engine. The actual task queue is entirely backed by Postgres, but things like streaming events between different workers are done through RabbitMQ at the moment, as well as sending a message from one component to another when you distribute the engine components. I've written a little more about this here: https://news.ycombinator.com/item?id=396439…

That makes sense, though a bit disappointing. One hope of using Postgres as a task queue is simplifying your overall stack. Having to host RabbitMQ partially defeats that. I'll stay tuned for the Postgres-backed messaging!

Re: Launch HN: Hatchet (YC W24) – Open-source task queue, now with a cloud version

#29

Post by abelanger »

fangpenlin wrote: post #24

Hatchet looks pretty awesome. I was thinking about using it to replace my Celery worker. However, the problem is that I can only use the gRPC client to create a task (correct me if I am wrong). What I want is to be able to commit a bunch of database rows altogether with the background task itself directly. The benefit of doing so with a PostgreSQL database is that all the rows will be in the same transaction. With tr…

That's correct, you can only create tasks via the gRPC client, Hatchet can't hook into the same transaction as your inserts or updates.

It seems like a very lightweight tasks table in your existing PG database which represents whether or not the task has been written to Hatchet would solve both of these cases. Once Hatchet is sent the workflow/task to execute, it's guaranteed to be enqueued/requeued. That way, you could get the other benefits of Hatchet with still getting transactional enqueueing. We could definitely add this for certain ORM frameworks/SDKs with enough interest.

Post reply on HN