261–270 of 336 posts
Since we are at the topic, what the best way to schedule production grade tasks on a Windows server? I joined a new company and they are using Task Scheduler and it is just awful!
step 1. install Linux
I'm not sure what you mean when you say there is no easy solution... procure a small vm on Linode or your choice of provider for like $20/mo. SSH into it, set up the crontab to run your web scraping script. I could literally have that set up and running in 15 minutes flat. EDIT: after reading the other solutions in this thread I'm blown away by how much people are over thinking and over engineering this. KISS people.
Overthinking it? Your solutions costs $240/year. There are solutions out there that would probably bring that under $20/year, and be more reliable than your VM that falls over when it's full of logs (at some indeterminate future date). 10x+ savings with greater reliability isn't "overthinking it", it's a greatly improved solution.
If this is an issue, I think all other solutions have similar risks: account banned, company acquired/closed down, forced update to new API/system, to a point that I think I'd take the bet on a VM living longer in most cases
Raspberry Pi at home if you don't have a home server already. Or some low spec cheap vps.
Although they still seem to be out of stock almost everywhere. (Looking for a Zero W)
https://www.abstracloud.com/jobs It is definitely the easiest way for Python code. Just paste your code and thats it Full disclosure: I made this product. We are a YC company
Basic plan is $35/m which is much higher than alternatives discussed in this thread, isn't it?
All free: * free-tier vps on GCP or Oracle Cloud * lambda job on AWS I have a cheap VPS I use for other things and just run my cron jobs there.
The free tier on Oracle Cloud is nuts. 200GB block storage, 10gb object storage, 24GB RAM 4 core ARM VM, 2 x86 VM's 1GB RAM each. From my experience, it's also really hard to get charged since you have to upgrade your account from a free tier. I haven't been charged one cent in several years.
> It is 2022, so I had thought it would be tremendously easy and cheap, but it seems no solution is easily implemented. You assume someone purchased hardware (preferably available anywhere on the planet), powered it up, set it up, connected it to the internet, then built the software to handle your very specific task, then put it online to do specifically what you want and for a close-to-free price? And you're shocke…
There are two ways to set up a cron job: the dashboard or via the API if you need the flexibility, e.g., dynamically creating/starting/stopping crons.
We use HTTP webhooks to make the outgoing cron requests, which makes it easy to integrate with a wide range of services and platforms, including Vercel, Lambda, etc. We're also working on adding local execution so you can have even more control over how your tasks are run without using webhooks, but that's still being built & tested.
Feel free to reply here or email me directly if you have any questions: james@mergent.co
Mergent.io has this feature. Don't know if it's the cheapest.
- James