Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The State of Web Scraping 2022

scrapeops.io

Post reply on HN
Print view View original

121–130 of 145 posts

Re: The State of Web Scraping 2022

#121

Post by fleddr »

autoexec wrote: post #112
fleddr wrote: post #27

Earlier quoted context omitted.

No they don't, Google and Bing respect robots.txt. Most websites would open it up to them because they need the traffic, so it's a type of scraping that is beneficial. Any other scraping, especially when ignoring robots.txt, is unsolicited. And if said website takes additional advanced anti-scraping measures, and you persist in bypassing that too, then to me you're clearly unethical, even if it's technically legal. "…

> Any other scraping, especially when ignoring robots.txt, is unsolicited. And if said website takes additional advanced anti-scraping measures, and you persist in bypassing that too, then to me you're clearly unethical, even if it's technically legal. I suppose it just comes to down to your own morals, but I see nothing at all unethical about scraping a site for personal use provided that it's done gently enough to…

You're really taking the most innocent stance possible on scraping.

The article talks of large scale scraping, which includes all kinds of bypassing tools, proxies, hardware, or commercial services that abstract this away.

This industrial scale level of scraping is not the same thing as you saving a local copy of 3 web pages. The scale is a million times bigger and for sure it will not be for personal use.

Re: The State of Web Scraping 2022

#122

Post by zamadatix »

ForHackernews wrote: post #119
digitcatphd wrote: post #72

Earlier quoted context omitted.

Good take, IMO ethically speaking we should not penalize scrapers themselves but do so based on their use. Scraping Facebook to make a clone of profiles shouldn’t be held to the same scrutiny of scraping Facebook to do an internal analysis of user demographics for research purposes.

Why should either be discouraged?

Cloning profiles is what seems likely wrong to me but I'm not sure how that being done via scraping or not should matter.

Re: The State of Web Scraping 2022

#123

Post by Terry_Roll »

LunaSea wrote: post #56
fleddr wrote: post #27

Earlier quoted context omitted.

No they don't, Google and Bing respect robots.txt. Most websites would open it up to them because they need the traffic, so it's a type of scraping that is beneficial. Any other scraping, especially when ignoring robots.txt, is unsolicited. And if said website takes additional advanced anti-scraping measures, and you persist in bypassing that too, then to me you're clearly unethical, even if it's technically legal. "…

> No they don't, Google and Bing respect robots.txt. They don't.

Correct.

I myself wrote a webserver, albeit a specialised one and for curiosity, I also created a few pages which were in no way accessible unless you knew its web address, there were no links to these pages from the home page or anything, I didn't even tell anyone about these webpages and yet in my logs, I could see those webpages were being spidered!

My robots.txt was setup as an instruction to proceed no further, so I think there is other feedback mechanisms guiding the spiders but I havent worked out if its from the web browser, or actual infrastructure like switches or routers.

Admittedly this was before HTTPS became common.

Re: The State of Web Scraping 2022

#124

Post by jeroenhd »

blantonl wrote: post #15

I fail to understand why Web Scraping isn't almost universally viewed as unethical and a terrible and nasty business practice. In almost all cases I view Web scraping as people who are trying to build businesses on top of other people's innovation and data. I know this isn't a popular opinion, so change my mind, but at the same time, I'm one of those business owners that fights with Web scraping constantly and my opi…

Scraping itself isn't universally unethical. Google and Bing scraping websites to make information easier to access is fine, and scraping and analysing government data is even better. Public data should be public, after all.

However, the disgusting data brokers that employ most of the custom scrapers, are usually unethical. That's why I don't trust any person or company that admits being involved professionally in "scraping", because most of the time that means "we collect personal information that got leaked elsewhere and sell them on".

Re: The State of Web Scraping 2022

#125

Post by Fatnino »

RobSm wrote: post #81
KieranMac wrote: post #16

As a lawyer whose primary focus is in web scraping, this article is in many ways misleading and inaccurate. While it is true that the Van Buren case is generally positive for web scraping, the overall legal landscape is still murky. The main battleground for web scraping legal issues is shifting from the CFAA to breach of contract and various state-law issues, including misappropriation, unjust enrichment, and trespa…

How many contracts google breaches scraping billions of pages every month?

Google doesn't have to proactively try very hard to ingest sites. If something is difficult for Google to scrape they don't sped loads of engineer hours on getting it to work. They just leave the site out and the webmaster there will quickly bend over backwards to make sure Google can scrape them. When something gets scraped into Google inadvertently it's because the website made not even the slightest effort to protect itself.

Re: The State of Web Scraping 2022

#126

Post by megous »

Ian_Kerins wrote: post #45
blantonl wrote: post #15

I fail to understand why Web Scraping isn't almost universally viewed as unethical and a terrible and nasty business practice. In almost all cases I view Web scraping as people who are trying to build businesses on top of other people's innovation and data. I know this isn't a popular opinion, so change my mind, but at the same time, I'm one of those business owners that fights with Web scraping constantly and my opi…

Some web scraping can be unethical, say for example if you are scraping a site solely to mirror their content and add zero value to the original content owner. However, there are a lot of web scraping use cases which are beneficial to the site being scraped and actually add value. Two examples: - Google: Ahrefs & SEMRush scrape Google so they can provide SEO analytics to companies looking to grow their companies. Goo…

> Some web scraping can be unethical, say for example if you are scraping a site solely to mirror their content and add zero value to the original content owner.

Archiving is unethical?

Re: The State of Web Scraping 2022

#128

Post by wumpus »

kevinsundar wrote: post #98
wumpus wrote: post #53

Earlier quoted context omitted.

web.archive.org has a CDX index, similar to Common Crawl. Since I use both of these archives together, I wrote this code to iron out the differences between them: https://github.com/cocrawler/cdx_toolkit

Hey! I was using your tool a couple months ago. It was super helpful for my project.

Thanks! I rarely hear from users, great to hear from you!

Re: The State of Web Scraping 2022

#129

Post by platelminto »

chirau wrote: post #106
jconley wrote: post #44

Earlier quoted context omitted.

I have a project that will be fueled by scraping. We should chat. :)

I do tons of scraping as well, let me know if you need extra hands.

Hey, I'm in the middle of building a large scraping application - would you mind if I asked you for some advice? Email in bio!

Re: The State of Web Scraping 2022

#130

Post by jasonfarnon »

KieranMac wrote: post #84
faizshah wrote: post #62

Earlier quoted context omitted.

Is there a good blog or something that tracks these cases?

Prof. Eric Goldman's blog is probably the #1 site historically on scraping and the law. I've contributed to it a few times. https://blog.ericgoldman.org/archives/2021/06/more-perspecti... The name of my firm is McCarthy Garber Law. I write about scraping there when I have time (which I rarely do)!

I agree that Eric's blog is great for getting updates on what's going on, and I've been following it for years. But he is very one-sided in his opinions about decisions, particularly on controversial issues like section 230. I have to remind myself he's an academic (though at a law school) and I'm not just reading some defense firm's memos.
Post reply on HN