Web Scraping 101 with Python
scrapingbee.com
Web Scraping 101 with Python
1–10 of 134 posts
Re: Web Scraping 101 with Python
#2fetching html and then parsing it navigating the parsed result (or with regexp) is what used to work 20 years ago.
These days, with all these reactive javascript frameworks you better skip to item number 5: headless browsing.
Also mind that Facebook, Instagram, ... will have anti-scraping measures in place. It's a race ;)
Re: Web Scraping 101 with Python
#3fetching html and then parsing it navigating the parsed result (or with regexp) is what used to work 20 years ago. These days, with all these reactive javascript frameworks you better skip to item number 5: headless browsing. Also mind that Facebook, Instagram, ... will have anti-scraping measures in place. It's a race ;)
This. Even relatively simple websites are much harder to parse today. I did a minor side project for a customer scraping some info and anti-scraping measures were in full force. It feels like an all out war.
Re: Web Scraping 101 with Python
#4fetching html and then parsing it navigating the parsed result (or with regexp) is what used to work 20 years ago. These days, with all these reactive javascript frameworks you better skip to item number 5: headless browsing. Also mind that Facebook, Instagram, ... will have anti-scraping measures in place. It's a race ;)
I've found that a lot of the time that's not needed. Often you'll find the data as a JSON blob in the page and can just read it directly from there. Or find that there's an API endpoint that the javascript reads.
Re: Web Scraping 101 with Python
#5fetching html and then parsing it navigating the parsed result (or with regexp) is what used to work 20 years ago. These days, with all these reactive javascript frameworks you better skip to item number 5: headless browsing. Also mind that Facebook, Instagram, ... will have anti-scraping measures in place. It's a race ;)
This. Even relatively simple websites are much harder to parse today. I did a minor side project for a customer scraping some info and anti-scraping measures were in full force. It feels like an all out war.
Such as? I've never encounter anything I wasn't able to overcome.
Re: Web Scraping 101 with Python
#6PyPpeteer might be worth a look as well. Basically a port of the JS puppeteer project that drives headless Chrome via the Devtools API.
As mentioned elsewhere, using anything other than headless isn't useful beyond a fairly narrow scope these days.
Re: Web Scraping 101 with Python
#7Earlier quoted context omitted.
This. Even relatively simple websites are much harder to parse today. I did a minor side project for a customer scraping some info and anti-scraping measures were in full force. It feels like an all out war.
Such as? I've never encounter anything I wasn't able to overcome.
Usually starts from simple to difficult. User agent stuff, IP address detection, aggressive rate limiting, captcha checking, browser fingerprinting, etc.
Re: Web Scraping 101 with Python
#8It’s fun to combine jupyter notebooks and py scraping. If you are working 15 pages/screens deep, you can “stay at the coal face” and not have to rerun the whole script after making a change to the latest step.
Re: Web Scraping 101 with Python
#9Earlier quoted context omitted.
This. Even relatively simple websites are much harder to parse today. I did a minor side project for a customer scraping some info and anti-scraping measures were in full force. It feels like an all out war.
Such as? I've never encounter anything I wasn't able to overcome.
It can be overcome and, admittedly, I am new to this so for me that means way more time spent trying to make it work. The odd one that I got stuck on for a while was a presented list where individual record held pertinent details, but the list had, seemingly randomly, items that looked like records on the surface, but were not ( so those had to be identified and ignored ). Small things like that.
Still, I would love to learn more about your approach if you would be willing to share.
Re: Web Scraping 101 with Python
#10PyPpeteer might be worth a look as well. Basically a port of the JS puppeteer project that drives headless Chrome via the Devtools API. As mentioned elsewhere, using anything other than headless isn't useful beyond a fairly narrow scope these days. https://github.com/pyppeteer/pyppeteer
I think you'd be surprised by the amount of website you can scrape without an headless browser.
Even Google SERP can be scraped with a simple HTTP client.