What's the best way to get around AWS/Azure/... ip range ban and VPN ban when scrapping?
The State of Web Scraping in 2021
51–60 of 132 posts
Re: The State of Web Scraping in 2021
#52What kind of stuff are people needing to scrape?
I would expect it's roughly the same answers, just varying in the specifics:
* those which don't offer a _reasonable_ API, or (I would guess a larger subset) those which don't expose all the same information over their API
* those things which one wishes to preserve (yes, I'm aware that submitting them to the Internet Archive might achieve that goal)
* and then the subset of projects where it's just a fun challenge or the ubiquitous $other
As an example answer to your question, some sites are even offering bounties for scraped data, so one could scratch a technical itch and help data science at the same time:
Re: The State of Web Scraping in 2021
#53What's the best way to get around AWS/Azure/... ip range ban and VPN ban when scrapping?
There are other providers. I think big time scrapers use residential IPs
Re: The State of Web Scraping in 2021
#54Earlier quoted context omitted.
so will Google's freezing of the UA lead to less ability to web scrape for the non big company scrapers out there?
What? It’s just a text string in the header. How in the world would that possibly make it more difficult to scrape? All Chrome is doing is stop appending the current semver in the UA it sends.
I mean I realize it probably isn't a problematic, just wondering, but on the other hand it shouldn't be so difficult to follow the reasoning based on the context I would think:
poster says - in order to be able to scrape effectively you should appear to be a real human, use different UAs etc.
So as this change happens different UAs become one less thing that you can easily change to seem less suspicious, as a non-frozen UA would then be a suspicious sign after some time.
So a sort of side effect.
Re: The State of Web Scraping in 2021
#55What's the best way to get around AWS/Azure/... ip range ban and VPN ban when scrapping?
[deleted]
Re: The State of Web Scraping in 2021
#56I've been working on a scraping project in Scrapy over the last month, using Selenium as well. My Python skills are mediocre (mostly a Java/Kotlin dev).
Not only has it been a blast to try out, but also surprisingly easy to setup.
I now have around 11 domains being scraped 4 times a day through a well defined pipeline + ETL then pipes it to Firebase Firestore for consumption.
Next step is to write the page on top of it.
Re: The State of Web Scraping in 2021
#57Earlier quoted context omitted.
What? It’s just a text string in the header. How in the world would that possibly make it more difficult to scrape? All Chrome is doing is stop appending the current semver in the UA it sends.
I mean I realize it probably isn't a problematic, just wondering, but on the other hand it shouldn't be so difficult to follow the reasoning based on the context I would think: poster says - in order to be able to scrape effectively you should appear to be a real human, use different UAs etc. So as this change happens different UAs become one less thing that you can easily change to seem less suspicious, as a non-fro…
[deleted]
Re: The State of Web Scraping in 2021
#58[deleted]
Re: The State of Web Scraping in 2021
#59Earlier quoted context omitted.
What? It’s just a text string in the header. How in the world would that possibly make it more difficult to scrape? All Chrome is doing is stop appending the current semver in the UA it sends.
I mean I realize it probably isn't a problematic, just wondering, but on the other hand it shouldn't be so difficult to follow the reasoning based on the context I would think: poster says - in order to be able to scrape effectively you should appear to be a real human, use different UAs etc. So as this change happens different UAs become one less thing that you can easily change to seem less suspicious, as a non-fro…
[deleted]
Re: The State of Web Scraping in 2021
#60For Python, instead of BeautifulSoup I prefer to use selectolax which is 3-5 times faster. Also, I think very few people use MechanicalSoup nowadays. There are libraries that allow you to use headless Chrome, e.g. Playwright. It looks like the author of the article just googled some libraries for each language and didn't research the topic.
Lazyweb link: https://github.com/rushter/selectolax
although I don't follow the need to have what appears to be two completely separate HTML parsing C libraries as dependencies; seeing this in the readme for Modest gives me the shivers because lxml has _seen some shit_
> Modest is a fast HTML renderer implemented as a pure C99 library with no outside dependencies.
although its other dep seems much more cognizant about the HTML5 standard, for whatever that's worth: https://github.com/lexbor/lexbor#lexbor
---
> It looks like the author of the article just googled some libraries for each language and didn't research the topic
Heh, oh, new to the Internet, are you? :-D