Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Hatchet v1 – A task orchestration platform built on Postgres

github.com

Post reply on HN
Print view View original

31–40 of 78 posts

Re: Show HN: Hatchet v1 – A task orchestration platform built on Postgres

#31

Post by lysecret »

latchkey wrote: post #25

Cool project. Every time one of these projects comes up, I'm always somewhat disappointed it isn't an open source / postgres version of GCP Cloud Tasks. All I ever want is a queue where I submit a message and then it hits an HTTP endpoint with that message as POST. It is such a better system than dedicated long running worker listeners, because then you can just scale your HTTP workers as needed. Pairs extremely well…

Yea I also like this system only problem I was facing with it was http read will lead to timeouts/lost connections. And task queues specifically have a 30 min execution limit. But I really like how it separates the queueing logic from the whole application/execution graph. Task queues are one of my favourite pieces of cloud infrastructure.

Re: Show HN: Hatchet v1 – A task orchestration platform built on Postgres

#32

Post by lysecret »

krainboltgreene wrote: post #27

A lot of these tools show off what a full success backlog looks like, in reality I care significantly more about what failure looks like, debugging, etc.

Ha this is a really good point! I worked with so many different kinds of observability approaches and always fell back to traced logs. This might be part of the reason.

Re: Show HN: Hatchet v1 – A task orchestration platform built on Postgres

#35

Post by INTPenis »

followben wrote: post #34

How does this compare to other pg-backed python job runners like Procrastinate [0] or Chancy [1]? [0] https://github.com/procrastinate-org/procrastinate/ [1] https://github.com/TkTech/chancy

Celery also has postgres backend, but I maybe it's not as well integrated.

Re: Show HN: Hatchet v1 – A task orchestration platform built on Postgres

#36

Post by igor47 »

INTPenis wrote: post #35
followben wrote: post #34

How does this compare to other pg-backed python job runners like Procrastinate [0] or Chancy [1]? [0] https://github.com/procrastinate-org/procrastinate/ [1] https://github.com/TkTech/chancy

Celery also has postgres backend, but I maybe it's not as well integrated.

It's just a results backend, you still have to run rabbitmq or redis as a broker

Re: Show HN: Hatchet v1 – A task orchestration platform built on Postgres

#37

Post by igor47 »

lysecret wrote: post #29

This is awesome and I will take a closer look! One question: We ran into issue with using Postgres as a message queue with messages that need to be toasted/have large payloads (50mb+). Only fix we could find was using unlogged tables and a full vacuum on a schedule. We aren’t big Postgres experts but since you are I was wondering if you have fixed this issue/this framework works well for large payloads.

Don't put them in the queue. Put the large payload into an object store like s3/gcs and put a reference into the db or queue

Re: Show HN: Hatchet v1 – A task orchestration platform built on Postgres

#38

Post by igor47 »

latchkey wrote: post #25

Cool project. Every time one of these projects comes up, I'm always somewhat disappointed it isn't an open source / postgres version of GCP Cloud Tasks. All I ever want is a queue where I submit a message and then it hits an HTTP endpoint with that message as POST. It is such a better system than dedicated long running worker listeners, because then you can just scale your HTTP workers as needed. Pairs extremely well…

How do you deal with cloud tasks in dev/test?

Re: Show HN: Hatchet v1 – A task orchestration platform built on Postgres

#39

Post by wcrossbow »

followben wrote: post #34

How does this compare to other pg-backed python job runners like Procrastinate [0] or Chancy [1]? [0] https://github.com/procrastinate-org/procrastinate/ [1] https://github.com/TkTech/chancy

Another good one is pgqueuer https://github.com/janbjorge/pgqueuer

Re: Show HN: Hatchet v1 – A task orchestration platform built on Postgres

#40

Post by szvsw »

igor47 wrote: post #37
lysecret wrote: post #29

This is awesome and I will take a closer look! One question: We ran into issue with using Postgres as a message queue with messages that need to be toasted/have large payloads (50mb+). Only fix we could find was using unlogged tables and a full vacuum on a schedule. We aren’t big Postgres experts but since you are I was wondering if you have fixed this issue/this framework works well for large payloads.

Don't put them in the queue. Put the large payload into an object store like s3/gcs and put a reference into the db or queue

Yep - this is also the official recommended method by Hatchet, also sometimes called payload thinning.
Post reply on HN