The current article is better than nothing, but it missed https://playwright.dev
The State of Web Scraping in 2021
111–120 of 132 posts
Re: The State of Web Scraping in 2021
#112I think another technique that should be talked about is intercepting network responses as they happen. The web in 2021 still has a whole lot of client-side rendering. For those sites, data is often loaded on the fly with separate network calls (usually with some sort of nonce or contextual key). Much of the hassle in web scraping can be avoided by listening for that specific response instead of parsing an artifact of the JSON->JS->HTML process.
I put together a toy site [0] recently that uses this approach for JIT price comparisons of events. When you click on an event, the backend navigates to requested ticket provider pages through a pool of Puppeteer instances and waits for JSON responses with pricing data.
Re: The State of Web Scraping in 2021
#113What kind of stuff are people needing to scrape?
Building a side project using python scrapy to scrape podcast shows. I use it to search by title/description etc to find interesting podcasts. Also as a way to learn different tools and frameworks.
Re: The State of Web Scraping in 2021
#114Earlier quoted context omitted.
Interesting. I was about to start on some web automation and so far I've had hammered into my head that Selenium is the 'language of the internet' or something along those lines. What would be a better solution, if you have any to recommend?
I'd suggest Puppeteer / Playwright. Both are great. Iirc the puppeteer team largely moved to playwright.
Thank you for the suggestions! I will check them out.
Re: The State of Web Scraping in 2021
#115I've been working on a scraping project in Scrapy over the last month, using Selenium as well. My Python skills are mediocre (mostly a Java/Kotlin dev). Not only has it been a blast to try out, but also surprisingly easy to setup. I now have around 11 domains being scraped 4 times a day through a well defined pipeline + ETL then pipes it to Firebase Firestore for consumption. Next step is to write the page on top of…
Are you using Scrapy mainly for scraping, or do you do crawling, as well?
In my case I am only using it for direct scraping.
Re: The State of Web Scraping in 2021
#116Scraping things that don't want to be scraped is one of my favorite things to do. At work this is usually an interface for some sort of "network appliance." Though with the push for REST APIs over the last 6 years or so, I don't have a need to do it all to often. Plus with things like selenium it's too easy to just run the page as is, and I can't justify spending the time to figuring out the undocumented API. My favo…
I remember a workmate having to deal with some difficult to scrape data at a previous job - the page randomly rendered with different mark-up (but the same appearance) to mitigate pulling out data using selectors. I think he got to the bottom of it eventually but it made testing his work a pain.
Playwright's layout selectors might help the next time you encounter this.
https://playwright.dev/docs/selectors#selecting-elements-bas...
Re: The State of Web Scraping in 2021
#117Earlier quoted context omitted.
> Scraping things that don't want to be scraped If all else fails, no website can withstand OCR-based screen scraping. It is slow(er), but fast enough for many use cases.
Assuming that you eventually manage to load the page somehow. Which in some edge cases may entail simulating mouse movements and random delays.
Agreed. -> I use the ui.vision extension to simulate native mouse movements.
Re: The State of Web Scraping in 2021
#118Earlier quoted context omitted.
> Scraping things that don't want to be scraped If all else fails, no website can withstand OCR-based screen scraping. It is slow(er), but fast enough for many use cases.
Have you tried on a page protected by cloudflare captcha?
It seems to be no problem if you automate a real browser as opposed to a headless browser. I think they test for that.
Re: The State of Web Scraping in 2021
#119I love web scraping, and I used to download many images through a series of scripts that crawled throughout a certain website.
Re: The State of Web Scraping in 2021
#120On the Ruby side both Nokogiri and Mechanize should be mentioned...
Good call! ~ Will add them in the next version.