221–230 of 336 posts
Might be more precise to call it "GitHub scraping" as you set up your script to run on a schedule on GitHub Actions CI and keep its state by committing into the git repo.
We have our scripts running on GH Actions end by hitting the Slack API to notify us with a message.
You should check out Modal! Does much more than just cronjobs, but super easy and cheap: https://modal.com/docs/guide/cron . They have an example with a simple web scraper as well: https://modal.com/docs/guide/web-scraper From their pricing example: > You schedule a job to run once an hour for 15 seconds. This job uses 50% of one CPU core and 256 MiB of memory. This job will cost $0.016/day for the CPU, and $0.001/da…
If you are comfortable using AWS , You could us AWS Batch (free) with Spot Fargate jobs with Cloudwatch cron as the trigger . You only pay for the vcpu hours your job consumes https://aws.amazon.com/fargate/pricing/ , it’s a set it up once and forget it model. You could use lambda if your runtime is expected to always be less than 15 minutes Batch fargate jobs documentation - https://docs.aws.amazon.com/batch/latest/…
Earlier quoted context omitted.
I exclusively use Lambda with Cloudwatch Rules for all the crons I write these days. It just works out of the box.
You should check out EventBridge Scheduler. Just released last month and seems like the go-to solution for such things going forward. I especially like the "run once at this time in the future" feature.
Earlier quoted context omitted.
Overthinking it? Your solutions costs $240/year. There are solutions out there that would probably bring that under $20/year, and be more reliable than your VM that falls over when it's full of logs (at some indeterminate future date). 10x+ savings with greater reliability isn't "overthinking it", it's a greatly improved solution.
I charge my clients $550 an hour, so a $240 a year solution which takes 15 mins to set up is what I would choose over the time consuming alternatives offered.
https://www.abstracloud.com/jobs It is definitely the easiest way for Python code. Just paste your code and thats it Full disclosure: I made this product. We are a YC company