The State of Web Scraping 2022

Post by shukantpal » Wed, Jan 12, 2022, 8:30 PM UTC

RobSm wrote: post #89

fleddr wrote: post #27

Earlier quoted context omitted.

No they don't, Google and Bing respect robots.txt. Most websites would open it up to them because they need the traffic, so it's a type of scraping that is beneficial. Any other scraping, especially when ignoring robots.txt, is unsolicited. And if said website takes additional advanced anti-scraping measures, and you persist in bypassing that too, then to me you're clearly unethical, even if it's technically legal. "…

No. robots.txt is not something that is defined and enforced by the law. Just because someone came up with some 'recommendation' like robots.txt does not mean this is the law

Post by LamaOfRuin » Wed, Jan 12, 2022, 9:05 PM UTC

RobSm wrote: post #88

travisporter wrote: post #19

Earlier quoted context omitted.

Don’t get my home address, name, family members names, salary, cell phone number, aggregate and sell them and claim “it’s all publically available anyway”

If you post that data on a public domain, that is publicly available. It's like writing that info on a cardboard and putting it in the town square and then saying 'why you people steal my data!'

Post by tomschwiha » Wed, Jan 12, 2022, 9:15 PM UTC

stef25 wrote: post #83

senko wrote: post #68

Earlier quoted context omitted.

From https://developers.google.com/search/docs/advanced/robots/in... > [...] Googlebot and other respectable web crawlers obey the instructions in a robots.txt file [...] If you're saying this is a lie, please provide sources

On an eCommerce site I'm responsible for I changed some links from a GET to a POST. "BingPreview" continued hitting those links with GET requests, polluting my logs with 100s of "method not allowed" entries. So I blocked that UA from those links, nothing changed. Banned the bot all together, still hitting my site. This went on for well over a year.

Post by cookie_monsta » Wed, Jan 12, 2022, 9:49 PM UTC

tyingq wrote: post #95

cookie_monsta wrote: post #94

Earlier quoted context omitted.

Maybe if all the useful content on your site can fit into a snippet I don't want to visit it?

Maybe the useful content is something you don't know is there, so you settle for what's in the snippet. Because you imagine Google's AI surely extracted the right bits. There's also a sort of diminishing returns effect here. If google trains people that the snippet is good enough, less traffic goes to the site. Eventually, enough to shutter the site, for some sites. Then nobody has the info. The pattern has already a…

Post by tyingq » Wed, Jan 12, 2022, 9:55 PM UTC

cookie_monsta wrote: post #107

tyingq wrote: post #95

Earlier quoted context omitted.

Maybe the useful content is something you don't know is there, so you settle for what's in the snippet. Because you imagine Google's AI surely extracted the right bits. There's also a sort of diminishing returns effect here. If google trains people that the snippet is good enough, less traffic goes to the site. Eventually, enough to shutter the site, for some sites. Then nobody has the info. The pattern has already a…

Yes, it would be preferable if people were more curious and willing to explore topics in depth. But sometimes all you want to know is what's the capital of Moldavia. Ideally the web would be about easy access to relevant information, not a competition for harvesting page views.