1–10 of 11 posts
If yes, do they block beyond a certain number of hits? Is there a paid route to doing this at scale?
Are there popular libraries/packages that let us do this, that are current with T&C of these platforms?
This question is way too broad. What is your purpose? What specifically are you scraping, (ie images, text, audio, video)? Please expand
If all aren't possible, which are? Want to let my users derive searchable intelligence off this info - intent is commercial.
This question is way too broad. What is your purpose? What specifically are you scraping, (ie images, text, audio, video)? Please expand
I want to scrape all possible items of a post: video or photo when that's there, the caption, the hashtags, location tag. If all aren't possible, which are? Want to let my users derive searchable intelligence off this info - intent is commercial.
Some of the social platforms have APIs you can pay to access. Some have aggressive anti-scraping countermeasures.
Possible, very much so, just depends on the platform and the rate of access that they allow. Some platforms will basically rate limit hard if they detect a lot of traffic from a single IP.
With paid API access, you may have a higher rate available, and an easier time getting the data (usually without you have to parse HTML)
Your lawyer is the best opinion regarding legality.
Good luck.
There is a service called SerpAPI that provides an API around stuff you might scrape. Haven't tried it myself but heard its good.
they's on the archive dot org
that site has everything but their search is shite
so to find scraped things that they scraped, you need to scrape their site and build a non-broken search engine for yourself
but you'll find your post-scraped social media sites
and many other interdasting things