Make your data accessible or you’re fired
1–3 of 3 posts
Re: Make your data accessible or you’re fired
#2Web scraping is such an underappreciated art. Unfortunately, it's harder and harder to do little hobby projects since many sites will quickly ban you.
I was working on a music project, trying to use Rate Your Music as the data source. Got banned before even hitting 50 requests (less than one a minute). I should have checked their TOS first ...
Re: Make your data accessible or you’re fired
#3Web scraping is such an underappreciated art. Unfortunately, it's harder and harder to do little hobby projects since many sites will quickly ban you. I was working on a music project, trying to use Rate Your Music as the data source. Got banned before even hitting 50 requests (less than one a minute). I should have checked their TOS first ...
Some secrets:
- use a VPS (throw away IP) - rate limit yourself - cache all the things