41–50 of 76 posts
I would be really interested in a distributed job running system with a scheduler. I want a UI where I can set jobs via a schedule that run on one of many workers. In that same UI I want to be able to look at failed jobs and see all of the output from that job. I don't want to have to ssh over to the box where the job ran to retrieve a log file. I once had a job where we used Tidal Enterprise Scheduler which poorly i…
Does anyone know of a free service that lets you run a few scheduled CURL requests each week?
This is definitely a needed service. As a business, if something isn't core to the business model, it's better to pay money and avoid distractions, at least until it's clear that something is actually core to the business. That said, I've wanted something akin to a distributed cron, without the complexities of workflow engines like airflow or azkaban, so I started writing a little HTTP API for scheduling jobs [1], wh…
Sure, but what about avoiding liabilities?
Hello all! I built Posthook as a simple solution for web applications to schedule one-off tasks. It lets you schedule a request back to your application for a certain time, with an optional JSON payload. It can be an alternative to running your own systems like Sidekiq, Celery, or Quartz and the operational overhead that comes along with them. Cron jobs and cloud provider tools like CloudWatch Events are also used fo…
Neat! We've been doing something like this at work, and another simple solution (curious if your Google cloud infrastructure is built on something similar) is Azure Service Bus queues with a function app trigger. Unlike AWS SQS, Azure SB queues can have items scheduled to be queued at an arbitrary time, see https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.... . SQS can only delay up to 15 minutes, so you h…
Earlier quoted context omitted.
Thank you, good luck to you as well! I've found that with scheduling tasks for the future it's important to do a final check before fulfilling the action, whether it's sending an email, push notification, etc. You don't want to send a reminder for an event that has been deleted, for example. That is why I have decided to keep the scope small and let developers make the final decision there. Reports and retries are de…
I also think there might be a market for this kind of product .. Great to see more people think on the same line .. Got some paid signups from last week also .. You might be right about email/push, but it certainly comes handy if minimal config is required.
This started out as a distributed cron over 20 years ago, available as a saas product for a few years already.
Earlier quoted context omitted.
Does Posthook support recurring scheduling (i.e. every 15 minutes)? Also, is there retry logic? I.E. 3 retries, delay 60 seconds between retries?
It does not support recurring scheduling at the moment. Right now the retry logic is just one retry 5 seconds after the first failure. At which point the hook gets set to a failed status and failure notifications get sent out. Retries are tricky because depending on how the job is implemented they can cause more harm than good. So I plan to refine that more based on customer feedback.
Ah, yes, there's nothing like bringing capacity back online only to have it crushed by all your customers retrying at the same time.
AWS got bit hard by this[3] but there's a blog post[1] about it, which is linked to by the docs for their client software[2].
[1] https://aws.amazon.com/blogs/architecture/exponential-backof...
[2] https://docs.aws.amazon.com/general/latest/gr/api-retries.ht...
[3] https://aws.amazon.com/message/5467D2/ ... basically DynamoDB is a fundamental service for AWS and had implemented some new streams features. This all appeared to be working, but they were running closer to capacity than intended, and when a cluster went out this caused a cascading failure.
And see this which linked to that RCA: https://blog.scalyr.com/2015/09/irreversible-failures-lesson...
Hello all! I built Posthook as a simple solution for web applications to schedule one-off tasks. It lets you schedule a request back to your application for a certain time, with an optional JSON payload. It can be an alternative to running your own systems like Sidekiq, Celery, or Quartz and the operational overhead that comes along with them. Cron jobs and cloud provider tools like CloudWatch Events are also used fo…
Almost all popular job processing libraries in all major frameworks support executing 1 off tasks with robust retry policies and even support recurring tasks.
The operational complexity is really just running the sidekiq or celery command in another Docker container or if you don't use Docker, then setting up 1 extra systemd unit file.
I once had a Celery / Flask app running on a $20 / month digitalocean server. It handled over 3 million background jobs per month and it also hosted my DB server, cache server and web front end.
On your pricing page, your highest tier supports up to 100,000 requests per month at $129/month. How much would it cost to do 3 million requests per month?
Also how would you set up custom retry policies?
I would be really interested in a distributed job running system with a scheduler. I want a UI where I can set jobs via a schedule that run on one of many workers. In that same UI I want to be able to look at failed jobs and see all of the output from that job. I don't want to have to ssh over to the box where the job ran to retrieve a log file. I once had a job where we used Tidal Enterprise Scheduler which poorly i…
Would you pay for this? How much and what pricing scheme would you prefer?
I guess the big point is, I'm looking to use this for regularly scheduled jobs, not just as a queuing service to spread load.
My email is in my profile if you want to get in touch.