Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Hatchet – Open-source distributed task queue

github.com

Post reply on HN
Print view View original

121–130 of 195 posts

Re: Show HN: Hatchet – Open-source distributed task queue

#121

Post by necovek »

doctorpangloss wrote: post #68
eska wrote: post #61

Earlier quoted context omitted.

Introducing another piece of software instead of using one you already use anyway introduces new failures. That’s hardly aesthetics. As a professional I’m allergic to statements like “you better start believing in X”. How can you even have objective discourse at work like that?

> Introducing another piece of software instead of using one you already use anyway introduces new failures. Okay, but we're talking about this on a post about using another piece of software. What is the rational for, well this additional dependency, Hatchet, that's okay, and its inevitable failures are okay, but this other dependency, RabbitMQ, which does something different, but will have fewer failures for some o…

There is some implicit context you are missing here.

Tools like hatchet are one less dependency for projects already using Postgres: Postgres has become a de-facto database to build against.

Compare that to an application built on top of Postgres and using Celery + Redis/RabbitMQ.

Also, it seems like you are confusing aesthetic with ergonomics. Since forever, software developers have tried to improve on all of "aesthetics" (code/system structure appearance), "ergonomics" (how easy/fast is it to build with) and "performance" (how well it works), and the cycle has been continuous (we introduce extra abstractions, then do away with some when it gets overly complex, and on and on).

Re: Show HN: Hatchet – Open-source distributed task queue

#122

Post by leafmeal »

Yanael wrote: post #112

Looks very promising. Recently, I built an asynchronous DAG executor in Python, and I always felt I was reinventing the wheel, but when looking for a resilient and distributed DAG executor, nothing was really meeting the requirements. The feature set is appealing. Wondering if adding/removing/skipping nodes to the DAG dynamically at runtime is possible.

a little late now, but I wonder if https://github.com/DataBiosphere/toil might meet your requirements

Re: Show HN: Hatchet – Open-source distributed task queue

#123

Post by krawczstef »

abelanger wrote: post #113
krawczstef wrote: post #92

Can you explain why you chose every function to take in context? https://github.com/hatchet-dev/hatchet/blob/main/python-sdk/... This seems like a lot of boiler plate to write functions with to me (context I created http://github.com/DAGWorks-Inc/hamilton ).

We did it because there are methods that should be accessed which don't map to `args` cleanly. For example, we let users call `context.log`, `context.done` (to determine whether to return on cancellation) or `context.step_output` (to dynamically access a parent's step output). Perhaps there's a more pythonic way to do this? Admittedly this is a pattern we adapted from Go.

you could just make them optional arguments that you inject if they're declared. Happy to chat more. With Hamilton we could actually build an alternative way to describe your API pretty easily...

Re: Show HN: Hatchet – Open-source distributed task queue

#124

Post by krawczstef »

kamikaz1k wrote: post #118
abelanger wrote: post #113

Earlier quoted context omitted.

We did it because there are methods that should be accessed which don't map to `args` cleanly. For example, we let users call `context.log`, `context.done` (to determine whether to return on cancellation) or `context.step_output` (to dynamically access a parent's step output). Perhaps there's a more pythonic way to do this? Admittedly this is a pattern we adapted from Go.

Probably just have it attached to self, like self.context But nbd IMHO

yep nbd, but :/

Re: Show HN: Hatchet – Open-source distributed task queue

#126

Post by lelanthran »

hinkley wrote: post #48

It’s been about a dozen years since I heard someone assert that some CI/CD services were the most reliable task scheduling software for periodic tasks (far better than cron). Shouldn’t the scheduling be factored out as a separate library? I found that shocking at the time, if plausible, and wondered why nobody pulled on that thread. I suppose like me they had bigger fish to fry.

Honestly, I'm doing something like that right now, just not in a position to show.

All I want is a simple way to specify a tree of jobs to run to do things like checkout a git branch, build it, run the tests, then install the artifacts.

Or push a new static website to some site. Or periodically do something.

My grug brain simply doesn't want to deal with modern way of doing $SHIT. I don't need to manage a million different tasks per hour, so scaling vertically is acceptable to me, and the benefits of scaling horizontally simply don't appear in my use cases.

Re: Show HN: Hatchet – Open-source distributed task queue

#128

Post by jsmeaton »

jerrygenser wrote: post #28

Something I really like about some pub/sub systems is Push subscriptions. For example in GCP pub/sub you can have a "subscriber" that is not pulling events off the queue but instead is an http endpoint where events are pushed to. The nice thing about this is that you can use a runtime like cloud run or lambda and allow that runtime to scale based on http requests and also scale to zero. Setting up autoscaling for wor…

https://cloud.google.com/tasks is such a good model and I really want an open source version of it (or to finally bite the bullet and write my own).

Having http targets means you get things like rate limiting, middleware, and observability that your regular application uses, and you aren’t tied to whatever backend the task system supports.

Set up a separate scaling group and away you go.

Re: Show HN: Hatchet – Open-source distributed task queue

#129

Post by throwawaymaths »

ako wrote: post #70
kcorbitt wrote: post #17

I love your vision and am excited to see the execution! I've been looking for exactly this product (postgres-backed task queue with workers in multiple languages and decent built-in observability) for like... 3 years. Every 6 months I'll check in and see if someone has built it yet, evaluate the alternatives, and come away disappointed. One important feature request that probably would block our adoption: one reason…

Funny how this is vision now. I started my career 29 years ago at a company that build exactly this, but based on oracle. The agents would run on Solaris, aix, vax vms, hpux, windows nt, iris, etc. Was also used to create an automated cicd pipeline to build all binaries on all these different systems.

Also basically has existed as an open source (pro version has web dashboard and complex task zoo) drop-in library (no sidecar dependencies outside of postgres) in Elixir for years called Oban.

Re: Show HN: Hatchet – Open-source distributed task queue

#130

Post by danielovichdk »

necovek wrote: post #121
doctorpangloss wrote: post #68

Earlier quoted context omitted.

> Introducing another piece of software instead of using one you already use anyway introduces new failures. Okay, but we're talking about this on a post about using another piece of software. What is the rational for, well this additional dependency, Hatchet, that's okay, and its inevitable failures are okay, but this other dependency, RabbitMQ, which does something different, but will have fewer failures for some o…

There is some implicit context you are missing here. Tools like hatchet are one less dependency for projects already using Postgres: Postgres has become a de-facto database to build against. Compare that to an application built on top of Postgres and using Celery + Redis/RabbitMQ. Also, it seems like you are confusing aesthetic with ergonomics. Since forever, software developers have tried to improve on all of "aesth…

"Since forever, software developers have tried to improve on all of "aesthetics" (code/system structure appearance), "ergonomics" (how easy/fast is it to build with) and "performance" (how well it works), and the cycle has been continuous"

Fast,easy,well,cheap is not a quality measure but it sure is a way to build more useless abstractions. You tell me which abstractions has made your software twice as effective.

Post reply on HN