Pyppetteer is feature complete and worth noting: https://github.com/pyppeteer/pyppeteer
Thanks! Updated the blog post to include it.
The State of Web Scraping in 2021
121–130 of 132 posts
Re: The State of Web Scraping in 2021
#122Earlier quoted context omitted.
To be fair selenium style scraping can take a lot of time to setup if you aren’t already familiar with the tooling, and the browser rendering apis are unintuitive and sometimes flat out broken.
I was surprised not to see selenium in this article. It is a common tool
You're absolutely right. It slipped my mind as I considered it more of a language-agnostic tool, and I organized the article as the provisioning of tools for all popular programming languages. That said, I added it to the post as a language-agnostic tool - thanks for the pointer!
Re: The State of Web Scraping in 2021
#123Earlier quoted context omitted.
Yeah, there are services that give you a range of IPs for a certain time.
Even changing IPs won't always work against an adversary with a global view of the Internet such as CloudFlare. CF has a view on a significant chunk of internet traffic across many sites and feeds that into some kind of heuristics/machine learning. Even if we assume that your behavior on the scraped website looks human-like, you may still get blocked or challenged because of your lack of traffic on other sites. The I…
You nailed it! I've also faced issues in the past with captchas, and elaborate bot detection mechanisms. It would also be helpful to mention that there are automatic captcha solvers to bypass security once one is detected. I am wondering if it is worthwhile to provide an addition to this post on how to improve the efficacy of scraping despite these roadblocks. The article is geared towards beginner scrapers that are just starting out so maybe it would be overkill? What do you think?
Re: The State of Web Scraping in 2021
#124The current article is better than nothing, but it missed https://playwright.dev
Thanks for the pointer! - Just included Playwright as a language agnostic tool.
Re: The State of Web Scraping in 2021
#125Great article. I've put online a discord server for sharing knowledge about web scraping, if some of you wants to join https://discord.gg/fwqhqrWhHW
Re: The State of Web Scraping in 2021
#126Puppeteer/Playwright are the easiest for me
Re: The State of Web Scraping in 2021
#127What kind of stuff are people needing to scrape?
Re: The State of Web Scraping in 2021
#128Earlier quoted context omitted.
> Scraping things that don't want to be scraped If all else fails, no website can withstand OCR-based screen scraping. It is slow(er), but fast enough for many use cases.
Have you tried on a page protected by cloudflare captcha?
I have not had to deal with that, but I have idly thought that it might be easier to pipe the audio version into google assistant or something, and see what it comes up with.
Re: The State of Web Scraping in 2021
#129self promotion: I launched my no-code scraping cloud software on ProductHunt last month after a year of testing with beta users: https://www.producthunt.com/posts/browse-ai
Here are a few comparisons if you're curious:
- https://www.browse.ai/vs/hexomatic
- https://www.browse.ai/vs/import-io
- https://www.browse.ai/vs/octoparse
- https://www.browse.ai/vs/oxylabs
- https://www.browse.ai/vs/parsehub
- https://www.browse.ai/vs/simplescraper
Re: The State of Web Scraping in 2021
#130For Java/Kotlin, HtmlUnit is generally pretty great.
But JavaScript not being properly executed in HtmlUnit
https://stackoverflow.com/questions/19646612/javascript-not-
being-properly-executed-in-htmlunit