Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The State of Web Scraping 2022

scrapeops.io

Post reply on HN
Print view View original

91–100 of 145 posts

Re: The State of Web Scraping 2022

#91

Post by RobSm »

akersten wrote: post #78
bobblywobbles wrote: post #61

Not a lawyer, but many terms of service prohibit interacting with their website in an automated fashion, as well as collecting their data. In my understanding, scraping a site with these terms already puts you in the wrong.

> many terms of service prohibit interacting with their website in an automated fashion, Ignoring the fact that I didn't agree to anything just by virtue of requesting a page from a webserver (and, your server sent me the data! ), that's such a meaningless phrase that it's certainly unenforceable. What is an automated fashion? Do I have to manually craft my HTTP request by hand-pulsing a voltage on an Ethernet cable,…

This is so exactly. People do not realize that when they use chrome to view website, chrome is their 'scraper'.

And the goal of webs craping is not to get illegal data, but to have efficiency and performance by not doing something manually but letting computer do the repetitive tasks. It's a productivity tool. You can't make something illegal just because it's an automation instead of 'manual' operation.

Re: The State of Web Scraping 2022

#92

Post by Seattle3503 »

KieranMac wrote: post #16

As a lawyer whose primary focus is in web scraping, this article is in many ways misleading and inaccurate. While it is true that the Van Buren case is generally positive for web scraping, the overall legal landscape is still murky. The main battleground for web scraping legal issues is shifting from the CFAA to breach of contract and various state-law issues, including misappropriation, unjust enrichment, and trespa…

Is there a good blog post or summary that I could read?

Re: The State of Web Scraping 2022

#93

Post by RobSm »

xrendan wrote: post #24
blantonl wrote: post #15

I fail to understand why Web Scraping isn't almost universally viewed as unethical and a terrible and nasty business practice. In almost all cases I view Web scraping as people who are trying to build businesses on top of other people's innovation and data. I know this isn't a popular opinion, so change my mind, but at the same time, I'm one of those business owners that fights with Web scraping constantly and my opi…

I think it really depends on the application of web scraping. (As someone who does, what is in my mind, ethical web scraping) - Scraping public information from government websites to do analysis: ethical, it's the public's data - Scraping to help some companies customers more effectively use that companies product, for example scraping a medical office's insurance claims to help them automate their insurance remitta…

And if you manually copy someone's data they worked hard to generate to go and resell, then it's ethical?

Re: The State of Web Scraping 2022

#94

Post by cookie_monsta »

tyingq wrote: post #50
yashasolutions wrote: post #41

Earlier quoted context omitted.

Google is web scrapper number one, as any search engine. Making web scrapping illegal mean making search engine illegal. You do not want information to be public and/or free? Put it under login and charge for it. You want to prevent people to reuse the data you publish to build other (potentially competitive) products, then use licensing and copyright, and the law. However, banning a technological mean because what a…

Google does do some things that aren't great for website owners too. Like "rich snippets", where they present the information from your page right to the end user, leaving that end user with no reason to visit your site. And, I imagine, lots of A/B testing geared toward exactly that...keeping them on Google-owned properties.

Maybe if all the useful content on your site can fit into a snippet I don't want to visit it?

Re: The State of Web Scraping 2022

#95

Post by tyingq »

cookie_monsta wrote: post #94
tyingq wrote: post #50

Earlier quoted context omitted.

Google does do some things that aren't great for website owners too. Like "rich snippets", where they present the information from your page right to the end user, leaving that end user with no reason to visit your site. And, I imagine, lots of A/B testing geared toward exactly that...keeping them on Google-owned properties.

Maybe if all the useful content on your site can fit into a snippet I don't want to visit it?

Maybe the useful content is something you don't know is there, so you settle for what's in the snippet. Because you imagine Google's AI surely extracted the right bits.

There's also a sort of diminishing returns effect here. If google trains people that the snippet is good enough, less traffic goes to the site. Eventually, enough to shutter the site, for some sites. Then nobody has the info.

The pattern has already affected Google referral traffic to Wikipedia. Pageviews for Wikipedia are roughly flat from 2012 to today, where they had marked growth prior. 2012 is when Google starting rolling out their knowledge graph that presented Wikipedia data directly.

Re: The State of Web Scraping 2022

#96

Post by btown »

KieranMac wrote: post #16

As a lawyer whose primary focus is in web scraping, this article is in many ways misleading and inaccurate. While it is true that the Van Buren case is generally positive for web scraping, the overall legal landscape is still murky. The main battleground for web scraping legal issues is shifting from the CFAA to breach of contract and various state-law issues, including misappropriation, unjust enrichment, and trespa…

Not a lawyer, but is it at least true that web scraping alone would now be significantly less likely to be a basis for federal criminal prosecution under the CFAA?

I'm often reminded of the fact that in https://en.wikipedia.org/wiki/United_States_v._Swartz the scraped party JSTOR did not desire to press civil charges, but due to the criminal component of the CFAA, this was out of their hands - and the story ended in the worst possible way.

If the current legal landscape at least better restricts disputes over web scraping to civil litigation, it may not be a huge change for how companies look at their risks, but it could make a huge difference for individuals caught in the crossfire.

Re: The State of Web Scraping 2022

#97

Post by slaymaker1907 »

RobSm wrote: post #88
travisporter wrote: post #19

Earlier quoted context omitted.

Don’t get my home address, name, family members names, salary, cell phone number, aggregate and sell them and claim “it’s all publically available anyway”

If you post that data on a public domain, that is publicly available. It's like writing that info on a cardboard and putting it in the town square and then saying 'why you people steal my data!'

I disagree because there is a difference between posting something publicly for humans and posting something publicly for bots/large scale analysis. I'm ok with my employer possibly being able to see whether I am looking for a new job or not on LinkedIn if that means they would need to have a human looking at my LinkedIn page. I am not ok with them training some ML algorithm to monitor my LinkedIn page to determine how likely I am to leave the company at all times.

Another danger is when public but not easily accessible data is able to deanonymize datasets which is probably the norm rather than the exception for anonymized datasets. Sure there are technical measures to make it better, but at the end of the day I think a lot of privacy is about respecting social boundaries and not breaking these protection measures even if technically possible. Most of the time, these measures are really about keeping honest people honest and not about stopping dedicated attackers.

Re: The State of Web Scraping 2022

#98

Post by kevinsundar »

wumpus wrote: post #53
mycall wrote: post #29

Earlier quoted context omitted.

I wish web.archive.org had an index by someone like common crawl. There is lots of great stuff on archive.org

web.archive.org has a CDX index, similar to Common Crawl. Since I use both of these archives together, I wrote this code to iron out the differences between them: https://github.com/cocrawler/cdx_toolkit

Hey! I was using your tool a couple months ago. It was super helpful for my project.

Re: The State of Web Scraping 2022

#99

Post by kevinsundar »

mycall wrote: post #29
ok_coo wrote: post #22

Time for me to advocate again for people to use Common Crawl. Please don't slam peoples' websites, look for alternatives before scraping. There are probably other, better options. APIs, data set downloads, etc. https://commoncrawl.org/

I wish web.archive.org had an index by someone like common crawl. There is lots of great stuff on archive.org

They do and its better than common crawl's by my testing.

Re: The State of Web Scraping 2022

#100

Post by KieranMac »

btown wrote: post #96
KieranMac wrote: post #16

As a lawyer whose primary focus is in web scraping, this article is in many ways misleading and inaccurate. While it is true that the Van Buren case is generally positive for web scraping, the overall legal landscape is still murky. The main battleground for web scraping legal issues is shifting from the CFAA to breach of contract and various state-law issues, including misappropriation, unjust enrichment, and trespa…

Not a lawyer, but is it at least true that web scraping alone would now be significantly less likely to be a basis for federal criminal prosecution under the CFAA? I'm often reminded of the fact that in https://en.wikipedia.org/wiki/United_States_v._Swartz the scraped party JSTOR did not desire to press civil charges, but due to the criminal component of the CFAA, this was out of their hands - and the story ended in…

Yes, I would agree with that first sentence. After Van Buren, web scraping alone would now be significantly less likely to be a basis for federal criminal prosecution under the CFAA.
Post reply on HN