191–200 of 336 posts
You should check out Modal! Does much more than just cronjobs, but super easy and cheap: https://modal.com/docs/guide/cron . They have an example with a simple web scraper as well: https://modal.com/docs/guide/web-scraper From their pricing example: > You schedule a job to run once an hour for 15 seconds. This job uses 50% of one CPU core and 256 MiB of memory. This job will cost $0.016/day for the CPU, and $0.001/da…
I'm not sure what you mean when you say there is no easy solution... procure a small vm on Linode or your choice of provider for like $20/mo. SSH into it, set up the crontab to run your web scraping script. I could literally have that set up and running in 15 minutes flat. EDIT: after reading the other solutions in this thread I'm blown away by how much people are over thinking and over engineering this. KISS people.
I use Serverless with Lambda. It's even a template in Serverless, so with the CLI you just type `serverless` and select the Node.js/Python Scheduled Task template and everything is set up for you. I do a lot of web scraping, and Lambdas are very useful for web scraping since their IPs can change for each request. For deployment, you can just do `serverless deploy` or set up a GitHub action each time you push to main.
Quoted post unavailable.
Alternatively, for a more generic solution to “easy free cronjobs”, you can write some php page on a free php hosting service and point visualping to that page. Cost: 0/year
I recently utilized Netlify for this. My build was actually my scraper and I used their Scheduled Functions (cron) to trigger this build every so often. https://www.netlify.com/blog/how-to-schedule-deploys-with-ne...
https://docs.netlify.com/cli/get-started/#get-started-with-n...
Most folks here are focusing on the scheduling part, but change detection means you need some way to persist state (or have a long running process that keeps state in memory). While you could do FaaS (e.g. AWS Lambda) + another service for state (e.g. S3/RDS/EFS), that seems like overkill. Even today I would still do it the old way: store state in a filesystem (maybe SQLite if you have more complex needs), and config…
https://docs.aws.amazon.com/glue/latest/dg/monitor-data-ware...
Earlier quoted context omitted.
Overthinking it? Your solutions costs $240/year. There are solutions out there that would probably bring that under $20/year, and be more reliable than your VM that falls over when it's full of logs (at some indeterminate future date). 10x+ savings with greater reliability isn't "overthinking it", it's a greatly improved solution.
Cheapest linode is $5/month. Not sure why you'd need one that is $20/month
My recommendation: - (free) set up google sheets (not sure if this is required but that's what I do) - (free) run an scheduled appscript bound to that sheet with regular intervals - make it scrape the page (it's just javascript, you can do get requests, etc) - Have it scrape a webarchive of the page you want to compare (not need to persist state or run a database, etc) and have it check the differences with the curre…
I think it's a great starting point for students because it's totally free, abstracts over the coplications of oauth (for small things I still prefer it because it's so effortless) and you can combine google apps in fun ways. Considering many students live out of gmail + google docs, it's great
And having a GUI version of the app makes interacting with other students easy bc they can always edit the google sheet (aka database), forms, gmail, etc
Earlier quoted context omitted.
Quoted post unavailable.
The default branch on GitHub has been main for over 2 years. https://github.blog/changelog/2020-10-01-the-default-branch-...