Pyppetteer is feature complete and worth noting: https://github.com/pyppeteer/pyppeteer
The State of Web Scraping in 2021
91–100 of 132 posts
Re: The State of Web Scraping in 2021
#92Scraping things that don't want to be scraped is one of my favorite things to do. At work this is usually an interface for some sort of "network appliance." Though with the push for REST APIs over the last 6 years or so, I don't have a need to do it all to often. Plus with things like selenium it's too easy to just run the page as is, and I can't justify spending the time to figuring out the undocumented API. My favo…
> Scraping things that don't want to be scraped If all else fails, no website can withstand OCR-based screen scraping. It is slow(er), but fast enough for many use cases.
Assuming that you eventually manage to load the page somehow. Which in some edge cases may entail simulating mouse movements and random delays.
Re: The State of Web Scraping in 2021
#93Last year I needed some quick scraping and I used a headless Chromium to render webpages and print the HTML then analyze it with C#. I don't remember exactly, but I think it was around 100 or 200 loc, so not exactly something that took long to write. In fact the most difficult thing was to figure how to pass the right args to Chromium. I wonder what does a scraping framework offer?
Throttling is a prime example. If you start loading multitudes of sites in asynchronous fashion you'll have to enter some delay otherwise you run the risk of choking the server in misconfigured sites. I've DDoSed sites accidentally this way. You can of course build a framework on your own, and that's pretty much what every scraper does eventually, but it takes time and a lot of effort.
Re: The State of Web Scraping in 2021
#94Does anyone have a resource for getting into app-based scraping, if the API is obfuscated or rate limited?
Run an http debugger, or some proxy, and find the endpoints.
Re: The State of Web Scraping in 2021
#95Scraping things that don't want to be scraped is one of my favorite things to do. At work this is usually an interface for some sort of "network appliance." Though with the push for REST APIs over the last 6 years or so, I don't have a need to do it all to often. Plus with things like selenium it's too easy to just run the page as is, and I can't justify spending the time to figuring out the undocumented API. My favo…
> Scraping things that don't want to be scraped If all else fails, no website can withstand OCR-based screen scraping. It is slow(er), but fast enough for many use cases.
A browser extension is probably an easier way to extract text than OCR (unless you're targeting a wide range of sites, I suppose).
Re: The State of Web Scraping in 2021
#96Scraping things that don't want to be scraped is one of my favorite things to do. At work this is usually an interface for some sort of "network appliance." Though with the push for REST APIs over the last 6 years or so, I don't have a need to do it all to often. Plus with things like selenium it's too easy to just run the page as is, and I can't justify spending the time to figuring out the undocumented API. My favo…
> Scraping things that don't want to be scraped If all else fails, no website can withstand OCR-based screen scraping. It is slow(er), but fast enough for many use cases.
Have you tried on a page protected by cloudflare captcha?
Re: The State of Web Scraping in 2021
#97For Python, instead of BeautifulSoup I prefer to use selectolax which is 3-5 times faster. Also, I think very few people use MechanicalSoup nowadays. There are libraries that allow you to use headless Chrome, e.g. Playwright. It looks like the author of the article just googled some libraries for each language and didn't research the topic.
requests-html is faster than bs4 using lxml. It's a wrapper over lxml. I built something similar years ago using a similar method, it was much faster than bs4, too.
Re: The State of Web Scraping in 2021
#98Cloudflare's protection is quite a b*tch to circumvent with any headless or python library.
It's a pain even when you aren't a bot. For a while there, Cloudflare's fingerprinting page would trigger Firefox on Linux to crash instantly.
Re: The State of Web Scraping in 2021
#99I've been working on a scraping project in Scrapy over the last month, using Selenium as well. My Python skills are mediocre (mostly a Java/Kotlin dev). Not only has it been a blast to try out, but also surprisingly easy to setup. I now have around 11 domains being scraped 4 times a day through a well defined pipeline + ETL then pipes it to Firebase Firestore for consumption. Next step is to write the page on top of…
Are you using Scrapy mainly for scraping, or do you do crawling, as well?
Re: The State of Web Scraping in 2021
#100Nowadays is more and more common for websites to have some kind of rate limiting middleware such as rack attack for ruby. It would be interesting to explore the strategies to deal with it.
> It would be interesting to explore the strategies to deal with it.
The strategy to deal with it is to behave well when making requests so that you don't get rate limited.