101–110 of 336 posts
GCP Cloud Run with a schedule set to call the HTTP endpoint once an hour. Cloud Run will run any docker image, and give it a HTTPS URL. Scales to zero, allows running in the background for up to an hour. The benefit over FAAS is that you can run anything you want in the container, including multiple processes.
AWS Lambda and Azure Functions both support timer triggers and probably round down to $0/ month if your job only runs for a few seconds an hour
I exclusively use Lambda with Cloudwatch Rules for all the crons I write these days. It just works out of the box.
AWS Lambda and Azure Functions both support timer triggers and probably round down to $0/ month if your job only runs for a few seconds an hour
Raspberry Pi at home if you don't have a home server already. Or some low spec cheap vps.
GitHub actions are another option
While you could do FaaS (e.g. AWS Lambda) + another service for state (e.g. S3/RDS/EFS), that seems like overkill.
Even today I would still do it the old way: store state in a filesystem (maybe SQLite if you have more complex needs), and configure a plain old cronjob on any of the servers you have access to.
Maybe put it in docker/dockerhub to make it easier to run.
Surprisingly it is an interesting question. You have a very simple task, but ideally you want the solution to include: running environment (let's say a container), scheduler, persistent state, monitoring (you want to know when it is down). Not to mention deployment.
There are solutions for each of those, but given the simplicity it would be nice to have a lightweight solution that includes all of that with minimal configuration. I doubt it exists.
I.e. you can do all of that on AWS, but I can't help but wonder if the infrastructure setup is going to be more complicated than your actual scraper.
Edit: a good way to think about it is: imagine that a person just learned how to scrape websites, they wrote a script that works locally. How much do they need to learn to move it to the cloud?
If your answer is: ok, so you just need to create an AWS account, dockerize your script, put it in a Lambda, create RDS instance, trigger your Lambda through EventBridge and setup CloudWatch... then there's something wrong with you. There must be a better way to reliably run a fucking 10 line script in the cloud.
There's been many times over the years when I wanted to write a simple scraper that does exactly what author describes and what stopped me is the amount of infrastructure bullshit I would have to deal with.
AWS Lambda and Azure Functions both support timer triggers and probably round down to $0/ month if your job only runs for a few seconds an hour
I wrote a notifier for when a local business has open appointments. A quick fetch every day and connected it to SNS seamlessly. Now I can easily add family members if they want the notifications and they can easily unsubscribe right from the email. All within the free tier.
Is it about how to run cron? This is a thing of the past. As you are asking for 2022, look at systemd timers.
Is it about cheap hosting? The are lots of answers here already.
Or is it about how to monitor a site for changes?