Not a lawyer, but many terms of service prohibit interacting with their website in an automated fashion, as well as collecting their data. In my understanding, scraping a site with these terms already puts you in the wrong.
The State of Web Scraping 2022
61–70 of 145 posts
Re: The State of Web Scraping 2022
#62As a lawyer whose primary focus is in web scraping, this article is in many ways misleading and inaccurate. While it is true that the Van Buren case is generally positive for web scraping, the overall legal landscape is still murky. The main battleground for web scraping legal issues is shifting from the CFAA to breach of contract and various state-law issues, including misappropriation, unjust enrichment, and trespa…
Is there a good blog or something that tracks these cases?
Re: The State of Web Scraping 2022
#63I fail to understand why Web Scraping isn't almost universally viewed as unethical and a terrible and nasty business practice. In almost all cases I view Web scraping as people who are trying to build businesses on top of other people's innovation and data. I know this isn't a popular opinion, so change my mind, but at the same time, I'm one of those business owners that fights with Web scraping constantly and my opi…
I think it really depends on the application of web scraping. (As someone who does, what is in my mind, ethical web scraping) - Scraping public information from government websites to do analysis: ethical, it's the public's data - Scraping to help some companies customers more effectively use that companies product, for example scraping a medical office's insurance claims to help them automate their insurance remitta…
Re government websites: they're often terrible. I've occasionally contemplated a side project just to scrape and restructure some local/state websites into a usable forms with search and whatnot.
Re: The State of Web Scraping 2022
#64I fail to understand why Web Scraping isn't almost universally viewed as unethical and a terrible and nasty business practice. In almost all cases I view Web scraping as people who are trying to build businesses on top of other people's innovation and data. I know this isn't a popular opinion, so change my mind, but at the same time, I'm one of those business owners that fights with Web scraping constantly and my opi…
Because there would no Internet search - no search engines, no Google Search, and essentially no Internet bigger than a hobbyist DARPA - without web scraping.
Re: The State of Web Scraping 2022
#65Earlier quoted context omitted.
Google is web scrapper number one, as any search engine. Making web scrapping illegal mean making search engine illegal. You do not want information to be public and/or free? Put it under login and charge for it. You want to prevent people to reuse the data you publish to build other (potentially competitive) products, then use licensing and copyright, and the law. However, banning a technological mean because what a…
Google does do some things that aren't great for website owners too. Like "rich snippets", where they present the information from your page right to the end user, leaving that end user with no reason to visit your site. And, I imagine, lots of A/B testing geared toward exactly that...keeping them on Google-owned properties.
That always struck me as unethical as well.
Re: The State of Web Scraping 2022
#66I'm scraping about 30 sites for work at the moment, but have a few that are using Cloudflare which has been a b*tch to deal with. Tried numerous libraries and different proxy providers, but reliability is patchy. Previous fixes like https://github.com/Anorov/cloudflare-scrape don't seem to work anymore after Cloudflare updates, so I've switched to using a pretty optimised headless browser with good proxies instead.
I'm finding that Cloudflare is even blocking my RSS reader from requesting feeds behind their service. It's not even just scrapers at this point.
Re: The State of Web Scraping 2022
#67My toolbox of choice for web scraping is either Nokogiri or puppeteer
Can someone sell me on beautiful soup or scrapy or any of the others? Do they provide any advantages or features that I'd be missing out on?
Re: The State of Web Scraping 2022
#68Earlier quoted context omitted.
No they don't, Google and Bing respect robots.txt. Most websites would open it up to them because they need the traffic, so it's a type of scraping that is beneficial. Any other scraping, especially when ignoring robots.txt, is unsolicited. And if said website takes additional advanced anti-scraping measures, and you persist in bypassing that too, then to me you're clearly unethical, even if it's technically legal. "…
> No they don't, Google and Bing respect robots.txt. They don't.
From https://developers.google.com/search/docs/advanced/robots/in...
> [...] Googlebot and other respectable web crawlers obey the instructions in a robots.txt file [...]
If you're saying this is a lie, please provide sources
Re: The State of Web Scraping 2022
#69I fail to understand why Web Scraping isn't almost universally viewed as unethical and a terrible and nasty business practice. In almost all cases I view Web scraping as people who are trying to build businesses on top of other people's innovation and data. I know this isn't a popular opinion, so change my mind, but at the same time, I'm one of those business owners that fights with Web scraping constantly and my opi…
> I fail to understand why Web Scraping isn't almost universally viewed as unethical and a terrible and nasty business practice.
Scraping is simply a way to get data. I used to run a team that was paid by large government contractors in the US to scrape their job posts from their career portals, and then deliver those posts via email, fax and snail mail to veteran's service officers near the job opening. It was required by regulation, and the only way to get the job data was to scrape.Many enterprise applicant tracking systems did not have a good way to automatically deliver that data or wanted $millions for that capability. Scraping was the best way and in some cases, the only way.
By the way, search engines like Google are scrape data and index it.
Re: The State of Web Scraping 2022
#70I fail to understand why Web Scraping isn't almost universally viewed as unethical and a terrible and nasty business practice. In almost all cases I view Web scraping as people who are trying to build businesses on top of other people's innovation and data. I know this isn't a popular opinion, so change my mind, but at the same time, I'm one of those business owners that fights with Web scraping constantly and my opi…
Google is web scrapper number one, as any search engine. Making web scrapping illegal mean making search engine illegal. You do not want information to be public and/or free? Put it under login and charge for it. You want to prevent people to reuse the data you publish to build other (potentially competitive) products, then use licensing and copyright, and the law. However, banning a technological mean because what a…
Google is a crawler not a scraper, these are two totally different things