What kind of stuff are people needing to scrape?
Websites which change over time and don’t provide a simpler way of getting an update (e.g. an RSS feed or a JSON api).
The State of Web Scraping in 2021
71–80 of 132 posts
Re: The State of Web Scraping in 2021
#72What's the best way to get around AWS/Azure/... ip range ban and VPN ban when scrapping?
The large proxy providers operate in a sort of gray market. You pay for "residential" or "ISP" based IP addresses. In some instances these proxy connections are literally being tunneled through browser extensions running on a real world system somewhere (https://hola.org/ for instance)
Re: The State of Web Scraping in 2021
#73Cloudflare's protection is quite a b*tch to circumvent with any headless or python library.
https://news.ycombinator.com/item?id=28514998#28515629
> Cloudflare's bot protection mostly makes use of TLS fingerprinting, and thus pretty easy to bypass.
https://news.ycombinator.com/item?id=28251700 -> https://github.com/refraction-networking/utls
Disclaimer: haven't tried it.
Re: The State of Web Scraping in 2021
#74Cloudflare's protection is quite a b*tch to circumvent with any headless or python library.
with node, i've had success with puppeteer-extra using puppeteer-extra-plugin-stealth
Re: The State of Web Scraping in 2021
#75In my own experience puppeteer is much better/capable than selenium but the problem is that puppeteer requires nodejs. its python-wrapper https://github.com/pyppeteer/pyppeteer was not as good as selenium when you like to use python.
Re: The State of Web Scraping in 2021
#76Self promotion: my SaaS is the lowest cost web scraping tool for high volume, and has been in business since 2016.
My SaaS requires some technical knowledge to use (call a web api) which I suppose is why it's not ever in these lists.
Some of my customers are *very* large businesses. If you are looking at evading bot countermeasures, my product isn't probably the best for you. but for TCO nothing beats it.
Re: The State of Web Scraping in 2021
#77Self promotion: my SaaS is the lowest cost web scraping tool for high volume, and has been in business since 2016. https://PhantomJsCloud.com My SaaS requires some technical knowledge to use (call a web api) which I suppose is why it's not ever in these lists. Some of my customers are *very* large businesses. If you are looking at evading bot countermeasures, my product isn't probably the best for you. but for TCO no…
Isn't phantomjs deprecated and unmaintained?
Re: The State of Web Scraping in 2021
#78Self promotion: my SaaS is the lowest cost web scraping tool for high volume, and has been in business since 2016. https://PhantomJsCloud.com My SaaS requires some technical knowledge to use (call a web api) which I suppose is why it's not ever in these lists. Some of my customers are *very* large businesses. If you are looking at evading bot countermeasures, my product isn't probably the best for you. but for TCO no…
Isn't phantomjs deprecated and unmaintained?
Yep, according to PhantomJS' README, their "development is suspended until further notice".
It looks like phantomjscloud.com also supports Puppeteer.
Re: The State of Web Scraping in 2021
#79Scraping things that don't want to be scraped is one of my favorite things to do. At work this is usually an interface for some sort of "network appliance." Though with the push for REST APIs over the last 6 years or so, I don't have a need to do it all to often. Plus with things like selenium it's too easy to just run the page as is, and I can't justify spending the time to figuring out the undocumented API. My favo…
To be fair selenium style scraping can take a lot of time to setup if you aren’t already familiar with the tooling, and the browser rendering apis are unintuitive and sometimes flat out broken.
Maybe it's because I'm using the python bindings, but it took me about an hour to go from never using it to having it do what I needed it to do. I just messed around in a jupyter notebook until I got what I needed working. Tab complete on live objects is your friend. The hardest part was figuring out where to download a headless browser from.
Though I do prefer requests/bs4. I wrote a helper to generate a requests.Session object from a selenium Browser object. I had something recently where the only thing I needed the javascript engine for was a login form that changed. So by doing it this way I didn't have to rewrite the whole thing. Still kind of bothers me I didn't take the time to figure out how to do it without the headless browser, but it works fine, and I have other things to do.
Re: The State of Web Scraping in 2021
#80For Python, instead of BeautifulSoup I prefer to use selectolax which is 3-5 times faster. Also, I think very few people use MechanicalSoup nowadays. There are libraries that allow you to use headless Chrome, e.g. Playwright. It looks like the author of the article just googled some libraries for each language and didn't research the topic.
> It looks like the author of the article just googled some libraries for each language and didn't research the topic
Yep, this seemed like an aggregate Google results page.
I was initially intrigued by the article and then realized it was a list of libraries the author found via Google. There were significantly notable omissions from this list and a bunch of weird stuff that feels unnecessary. I don't think the author has actually scraped a page before.