The State of Web Scraping 2022
scrapeops.io
The State of Web Scraping 2022
1–10 of 145 posts
Re: The State of Web Scraping 2022
#2I'm scraping about 30 sites for work at the moment, but have a few that are using Cloudflare which has been a b*tch to deal with. Tried numerous libraries and different proxy providers, but reliability is patchy. Previous fixes like https://github.com/Anorov/cloudflare-scrape don't seem to work anymore after Cloudflare updates, so I've switched to using a pretty optimised headless browser with good proxies instead.
Re: The State of Web Scraping 2022
#3I'm scraping about 30 sites for work at the moment, but have a few that are using Cloudflare which has been a b*tch to deal with. Tried numerous libraries and different proxy providers, but reliability is patchy. Previous fixes like https://github.com/Anorov/cloudflare-scrape don't seem to work anymore after Cloudflare updates, so I've switched to using a pretty optimised headless browser with good proxies instead.
This has a lot of good info on how to cloudflare and others work, and more creative ways to bypass them if the easier options don't work https://incolumitas.com/2021/05/20/avoid-puppeteer-and-playw...
Re: The State of Web Scraping 2022
#4With the right combination of proxies, user agents and browsers, you can scrape every website. Even those that seem unscrapable.
:
This outcome was great news for web scrapers, as it means that so long as a websites has made their data public you are not in violation of the CFAA when you scrape the data even if it is prohibited in some other way (T&Cs, robots.txt, etc).
Just because you can, doesn't mean you should. It would be better I think if there was a treatment of the ethics here, rather than a seemingly "ra-ra go bots" attitude, as though the only consideration is commercial.
Re: The State of Web Scraping 2022
#5With the right combination of proxies, user agents and browsers, you can scrape every website. Even those that seem unscrapable. : This outcome was great news for web scrapers, as it means that so long as a websites has made their data public you are not in violation of the CFAA when you scrape the data even if it is prohibited in some other way (T&Cs, robots.txt, etc). Just because you can, doesn't mean you should.…
this sort of implies that the 'ethics' would end up meaning that you shouldn't scrape if it is not wanted, although I suppose there can be ethics or other than commercial requirements that mean that you should.
Re: The State of Web Scraping 2022
#6With the right combination of proxies, user agents and browsers, you can scrape every website. Even those that seem unscrapable. : This outcome was great news for web scrapers, as it means that so long as a websites has made their data public you are not in violation of the CFAA when you scrape the data even if it is prohibited in some other way (T&Cs, robots.txt, etc). Just because you can, doesn't mean you should.…
100% agree, when scraping it should always be done respectfully.
- If they provide a API, then use it.
- Don't slam a website, ideally spread it out over hours of the day when there target audience is least active (night time).
- If you can get cached data from somewhere that works, then use that.
Most developers are respectful and only scrape what they really need, not only from an ethical point of view but also a cost and resources point of view. Scraping data is resource intensive and proxy costs can quickly rise to $1,000-$10,000 per month. So most only scrape the minimum they need.
The other thing here as well, is that a lot of the most popular sites being scraped, are also massive scrapers themselves. The big ecommerce sites are being scraped, but they are also scraping their competitors too.
Re: The State of Web Scraping 2022
#7If anyone has anything else they think was missed or should be included then let me know!
Re: The State of Web Scraping 2022
#8With the right combination of proxies, user agents and browsers, you can scrape every website. Even those that seem unscrapable. : This outcome was great news for web scrapers, as it means that so long as a websites has made their data public you are not in violation of the CFAA when you scrape the data even if it is prohibited in some other way (T&Cs, robots.txt, etc). Just because you can, doesn't mean you should.…
You dont even need to do that, go overt plain sight in yer face and call yourself a search engine!
Re: The State of Web Scraping 2022
#9Like most here, I am very good at web scraping and automated form fills. I keep trying to figure out a profitable side project or business idea to make out of it and keep coming up with nothing that works.
Any good ideas?
Re: The State of Web Scraping 2022
#10With the right combination of proxies, user agents and browsers, you can scrape every website. Even those that seem unscrapable. : This outcome was great news for web scrapers, as it means that so long as a websites has made their data public you are not in violation of the CFAA when you scrape the data even if it is prohibited in some other way (T&Cs, robots.txt, etc). Just because you can, doesn't mean you should.…
You dont even need to do that, go overt plain sight in yer face and call yourself a search engine!
Haha I love that people forget how google/bing are out there scraping everything and anyone who scrapes anything for any other reason is a "bad guy".
You can get around some web scraping blockers by just setting your user agent as Googlebot too which I find funny...