111–120 of 195 posts
Earlier quoted context omitted.
Well, you just got an user. Love the concept of temporal, but i can't justify the overhead you need with infra to make it work for the upper guys... And the cloud offering is a bit expensive for small companies.
Do you know about the Temporal startup program? It gives enough credits to offset support fees for 2 years. https://temporal.io/startup
Can you explain why you chose every function to take in context? https://github.com/hatchet-dev/hatchet/blob/main/python-sdk/... This seems like a lot of boiler plate to write functions with to me (context I created http://github.com/DAGWorks-Inc/hamilton ).
I've been looking for this exact thing for awhile now. I'm just starting to dig into the docs and examples, and I have a question on workflows. I have an existing pipeline that runs tasks across two K8 clusters and share a DB. Is it possible to define steps in a workflow where the step run logic is setup to run elsewhere? Essentially not having an inline run function defined, and another worker process listening for…
How does this compare to River Queue ( https://riverqueue.com/ )? Besides the additional Python and TS client libraries.
I'm personally very excited about River and I think it fills an important gap in the Go ecosystem! Also now that sqlc w/ pgx seems to be getting more popular, it's very easy to integrate.
Looks pretty great! My biggest issue with Celery has been that the observability is pretty bad. Even if you use Celery Flower, it still just doesn’t give me enough insight when I’m trying to debug some problem in production. I’m all for just using Postgres in service of the grug brain philosophy. Will definitely be looking into this, congrats on the launch!
I case you’re stuck with Celery for a while: I was hit with this same problem, and solved it by adding a sidecar HTTP server thread to the Python workers that would expose metrics written by the workers into a multithreaded registry. This has been working amazingly well in production for over two years now, and makes it really straightforward to get custom metrics out of a distributed Celery app.
My only question is why did you call it Hatchet if it doesn't cut down on your logs? I'll show myself out.
Can you explain why you chose every function to take in context? https://github.com/hatchet-dev/hatchet/blob/main/python-sdk/... This seems like a lot of boiler plate to write functions with to me (context I created http://github.com/DAGWorks-Inc/hamilton ).
We did it because there are methods that should be accessed which don't map to `args` cleanly. For example, we let users call `context.log`, `context.done` (to determine whether to return on cancellation) or `context.step_output` (to dynamically access a parent's step output). Perhaps there's a more pythonic way to do this? Admittedly this is a pattern we adapted from Go.
But nbd IMHO
With NATS in the stack, what's the advantage over using NATS directly?
With regards to Nex -- it isn't fully stable and only supports Javascript/Webassembly. It's also extremely new, so I'd be curious to see how things stabilize in the coming year.
Earlier quoted context omitted.
It's only a few billion instructions on a decent sized server these days
Damn, I want one of these 100GHz CPUs you have, that sounds great. I think you mean million :)
My laptop can execute about 400 billion CPU instructions per second on battery.
That's about 10 billion instructions in 25ms.
Ihat's the CPU alone, i.e. not including the GPU which would increase the total considerably. Also not counting SIMD lanes as separate: The count is bona fide assembly language instructions.
It comes from cores running at ~4GHz, 8 issued instructions per clock, times 12 cores, plus 4 additional "efficiency" cores adding a bit more. People have confirmed by measurement the 8 instructions per clock is achievable (or close) in well-optimised code. Average code is more like 2-3 per cycle.
Only for short periods as the CPU is likely to get hot and thermally throttle even with its fan. But when it throttles it'll still exceed 1 billion in 25ms.
For perspective on how far silicon has come, the GPU on my laptop is reported to do about 14 trillion floating-point 32-bit calculations per second.